Image stabilization related methods and apparatus

ABSTRACT

Methods and apparatus for processing images captured by a camera device including multiple optical chains, e.g., camera modules, are described. Three, 4, 5 or more optical chains maybe used. Different optical chains capture different images due to different perspectives. Multiple images, e.g., corresponding to different perspectives, are captured during a time period and are combined to generate a composite image. In some embodiments one of the captured images or a synthesized image is used as a reference image during composite image generation. The image used as the reference image is selected to keep the perspective of sequentially generated composite images consistent despite unintentional came movement and/or in accordance with an expected path of travel. Thus, which camera module provides the reference image may vary over time taking into consideration unintended camera movement. Composite image generation may be performed external to the camera device or in the camera device.

RELATED APPLICATIONS

The present application claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/899,097 filed Nov. 1, 2013 which is herebyexpressly incorporated by reference in its entirety.

FIELD

The present application relates to image processing methods andapparatus and, more particularly, to methods and apparatus related toimage stabilization in still images and/or a sequence of images, e.g.,video or a burst of frames.

BACKGROUND

High quality digital cameras have to a large extent replaced filmcameras. However, like film cameras, with digital cameras much attentionhas been placed by the camera industry on the size and quality of lenseswhich are used on the camera. Individuals seeking to take qualityphotographs are often encouraged to invest in large bulky and oftencostly lenses for a variety of reasons. Among the reasons for usinglarge aperture lenses is their ability to capture a large amount oflight in a given time period as compared to smaller aperture lenses.Telephoto lenses tend to be large not only because of their largeapertures but also because of their long focal lengths. Generally, thelonger the focal length, the larger the lens. A long focal length givesthe photographer the ability to take pictures from far away.

In the quest for high quality photos, the amount of light which can becaptured is often important to the final image quality. Having a largeaperture lens allows a large amount of light to be captured allowing forshorter exposure times than would be required to capture the same amountof light using a small lens. The use of short exposure times can reduceblurriness especially with regard to images with motion. The ability tocapture large amounts of light can also facilitate the taking of qualityimages even in low light conditions. In addition, using a large aperturelens makes it possible to have artistic effects such as small depth offield for portrait photography.

Large lenses sometimes also offer the opportunity to support mechanicalzoom features that allow a user to optically zoom in or out and/or toalter the focal length of the lens which is important for framing ascene without the need to move closer or further from the subject.

While large lenses have many advantages with regard to the ability tocapture relatively large amounts of light compared to smaller lenses,support large zoom ranges, and often allow for good control over focus,there are many disadvantages to using large lenses.

Large lenses tend to be heavy requiring relatively strong and oftenlarge support structures to keep the various lenses of a camera assemblyin alignment. The heavy weight of large lenses makes cameras with suchlenses difficult and bulky to transport. Furthermore, cameras with largelenses often need a tripod or other support to be used for extendedperiods of time given that the sheer weight of a camera with a largelens can become tiresome for an individual to hold in a short amount oftime.

In addition to weight and size drawbacks, large lenses also have thedisadvantage of being costly. This is because of, among other things,the difficulty in manufacturing large high quality optics and packagingthem in a manner in which they will maintain proper alignment over aperiod of time which may reflect the many years of use a camera lensesis expected to provide.

A great deal of effort has been directed in the camera industry tosupporting the use of large camera lenses and packaging them in a waythat allows different lenses to be used in an interchangeable manner ona camera body. However, for the vast majority of camera users, thedrawbacks to cameras with large lenses means that camera users tend notto use large lenses with such lenses often being left to professionalsand/or photo enthusiasts willing to incur the expense and trouble ofbuying and using large lenses.

In fact, many camera owners who own cameras with large high qualitylenses often find themselves taking pictures with small pocket sizecameras, often integrated into other devices such as their cell phones,personal digital assistants or the like, simply because they are moreconvenient to carry. For example, cell phone mounted cameras are oftenmore readily available for use when an unexpected photo opportunityarises or in the case of a general family outing where carrying largebulky camera equipment may be uncomfortable or undesirable.

To frame a given scene from a given point, the focal length (hence size)of the lens depends on the size (area) of the image sensor. The smallerthe image sensor, the smaller the focal length and the smaller the lensrequired. With advances in sensor technology, it is now possible to makesmall sensors, e.g., 5×7 mm² sensors, with relatively high pixel count,e.g., 8 megapixels. This has enabled the embedding of relatively highresolution cameras in small devices such as cell phones. The smallsensor size (compared to larger cameras such as changeable lenssingle-lens reflex (SRL) cameras) enables small focal length lenseswhich are much smaller and lighter than larger focal length lensesrequired for cameras with larger sensors.

Cell phone mounted cameras and other pocket sized digital camerassometimes rely on a fixed focal length lens which is also sometimesreferred to as a focus-free lens. With such lenses the focus is set atthe time of manufacture, and remains fixed. Rather than having a methodof determining the correct focusing distance and setting the lens tothat focal point, a small aperture fixed-focus lens relies on a largedepth of field which is sufficient to produce acceptably sharp images.Many cameras, including those found on most cell phones, with focus freelenses also have relatively small apertures which provide a relativelylarge depth of field. There are also some high end cell phones that useauto focus cameras.

For a lens of a digital camera to be useful, it needs to be paired witha device which detects the light passing through the lens and convertsit to pixel (picture element) values. A megapixel (MP or Mpx) is onemillion pixels. The term is often used to indicate the number of pixelsin an image or to express the number of image sensor elements of adigital camera where each sensor element normally corresponds to onepixel. Multi-color pixels normally include one pixel value for each ofthe red, green, and blue pixel components.

In digital cameras, the photosensitive electronics used as the lightsensing device is often either a charge coupled device (CCD) or acomplementary metal oxide semiconductor (CMOS) image sensor, comprisinga large number of single sensor elements, each of which records ameasured intensity level.

In many digital cameras, the sensor array is covered with a patternedcolor filter mosaic having red, green, and blue regions in anarrangement. In such a filter based approach to capturing a color image,each sensor element can record the intensity of a single primary colorof light. The camera then will normally interpolate the colorinformation of neighboring sensor elements, through a process sometimescalled demosaicing, to create the final image. The sensor elements in asensor array using a color filter are often called “pixels”, even thoughthey only record 1 channel (only red, or green, or blue) of the finalcolor image due to the filter used over the sensor element.

While a filter arrangement over a sensor array can be used to allowdifferent sensor elements to capture different colors of light thusallowing a single sensor to capture a color image, the need to carefullyalign the filter area with individual pixel size sensor elementscomplicates the manufacture of sensor arrays as compared to arrays whichdo not require the use of a multi-color filter array.

While small focal length lenses paired with relatively high resolutionsensors have achieved widespread commercial success in cell phones andpocket cameras, they often leave their owners longing for better picturequality, e.g., picture quality that can only be achieved with a largerpixel area and a larger lens opening to collect more light.

Smaller sensors require smaller focal length lenses (hence smallerlenses) to frame the same scene from the same point. Availability ofhigh pixel count small sensors means that a smaller lens can be used.However, there are a few disadvantages to using smaller sensors andlenses. First, the small pixel size limits the dynamic range of thesensor as only a small amount of light can saturate the sensor. Second,small lenses collect less total light which can result in grainypictures. Third, small lenses have small maximum apertures which makeartistic effects like small depth of field for portrait pictures notpossible.

One application for cameras is video. In movie productions, stageproductions, studio noise productions and/or other applications wherehigh quality video is desired, cameras are often mounted on tracks alongwhich a camera can be rolled as it captures images. The track mountingarrangement avoids or reduces the risk of jerky movements that may occurin the case of handheld video or other cameras particularly where acamera is being moved altering the distance between the camera and theobject or objects in the scene being captured.

In view of the above discussion it should be appreciated that there is aneed for improved method or apparatus which can address the effect ofmotion of an image capture device, e.g., camera device. In particular itwould be desirable if motion of a camera device could be taken intoconsideration when generating an image or a sequence of images toeliminate or reduce the effect of unintended motion. While track systemscan provide for smooth camera motion, it would be desirable if imagestabilization techniques could be developed which would allow forhandheld cameras to provide video or capture images which allow for thegeneration of video with smooth intended motion without the undesirablemotion often associated with video produced by handheld devices. Itwould be desirable if methods and/or apparatus could be developed whichwould allow for a camera device, e.g., a handheld camera device, tocapture images and then for the camera device or another device to beable to processor one or more images to produce an image taking intoconsideration motion, e.g., motion between images due to cameramovement. It would be desirable if at least some of the methods allowedfor image stabilization and generation of one or more images without theneed for tracks and/or other complicated camera mounting arrangementsintended to limit camera motion to a smooth physical path of motion.While it would be desirable to address one or more of the abovediscussed problems it should be appreciated that any method or apparatuswhich could improve the image quality of an image or sequence of imagesgenerated from one or more images captured by a device which may besubject to motion, and especially unintentional motion, would bedesirable even if it did not address all of the above discussedproblems.

SUMMARY OF THE INVENTION

Various embodiments are directed to methods of generating one or moreimages, e.g., a sequence of images, using a camera including multipleoptical chains or a light field camera taking into consideration cameramotion. Camera motion is detected, e.g., tracked, by using sensors inthe camera device and/or by comparing one or more images captured by thecamera device.

Images are generated from multiple sets of image data, e.g., imagescaptured by different optical chains or portions of a light field camerasensor in a way which allows the image being generated to have one of aplurality of different points of view. The point of view is controlled,along with optional use of cropping, in the generation of an image in amanner that gives the appearance of images corresponding to sequentialtime periods being captured by a camera moving along a smooth track ofmotion. Thus, since the image generation process takes intoconsideration camera motion and the generation of a composite image isdone in a way that can reduce or minimize the effect of unintentionalmotion at least in some embodiments the image generation results in orfacilitates image stabilization.

In one embodiment while each image in a sequence corresponds to adifferent, e.g., sequential time period, the generated imagecorresponding to one time period in the sequence is generated fromcaptured image data which is captured by multiple different individualoptical chain modules operating in parallel during the time period towhich the individual generated image corresponds. As part of the imagegeneration process, the images captured by different optical chainmodules corresponding to an individual time period may be combined basedon a point of view that is determined based on camera motion. The motionmay, and in some embodiments is, detected through the use of a gyroscopeand/or accelerometers. The point of view used from one frame to the nextis selected in some embodiments to provide the appearance of aconsistent or smoothly changing point of view as opposed to relying on acenter portion of the camera device or center of a set of optical chainmodules as the point of view. Thus, the point of view used forcontrolling the combining process maybe different from the point of viewof the individual optical chain modules used to capture the image databeing combined. It should be appreciated that due to the multiplepoints-of-view corresponding to the images captured by different cameramodules, it is also possible to extrapolate and/or otherwise simulate avirtual point of view that could exist between the actual point of viewof two optical chains. Accordingly, if motion information indicates thatthe desired perspective is from such an in-between virtual referencepoint, it is possible to generate a reference image corresponding to thein-between point reference point providing a perspective which isbetween the perspective of two actual modules.

Thus, as part of the combining operation the point of view may bedetermined and adjusted as may be necessary to simulate a smooth trackof motion taking into consideration the images that may be captured bythe optical chains having different points of view. Image cropping maybe used as part of the combining and/or image data processing operationas well as point of view control to ensure that area included in theoutput video sequence remains relatively consistent and changesgradually over time as might be expected by smooth intentional cameramotion as opposed to inadvertent motion. As should be appreciatedinadvertent motion often takes the form of sudden or jerky cameramotion, that may be the result of the use of a handheld camera device.Such motion can be detected by sensing a position change within a timeperiod that exceeds an expected amount of motion in the case of a smoothintentional change. Other techniques for be used for detectinginadvertent motion as well or alternatively.

By using a large synthetic aperture, e.g., simulated aperture generatedby using multiple smaller apertures in combination and by outputting animage smaller than the maximum image size which may be captured, imageadjustments, in the form of cropping and altering the point of view usedfor generating an image, can be used to reduce or eliminate the effectof unintended motion as a camera device is moved along a path, e.g., apath which is intended to be smooth but may be jerky or subject tounintentional changes in the actual point of view of individual opticalchain modules as a result of unintended motion.

Various methods and apparatus of the present invention are directed tomethods and apparatus for obtaining some or all of the benefits of usingrelatively large and long lens assemblies without the need for largelens and/or long lens assemblies, through the use of multiple opticalchain modules in combination.

Furthermore, in at least some embodiments the benefits of using track orother camera support system can be simulated and/or at least partiallyobtained without the need for track or other complicated camerasupport/motion control systems.

Using the methods and apparatus of the present invention, a handheldcamera can provide improved video generation results than might beachieved without the use of the methods described herein.

Optical chain modules including, in some embodiments, relatively shortfocal length lenses which require relatively little depth within acamera are used in some embodiments. While use of short focal lengthlens can have advantages in terms of small lens width, the methods andapparatus of the present are not limited to the use of such lenses andcan be used with a wide variety of lens types. In addition, whilenumerous embodiments are directed to autofocus embodiments, fixed focusembodiments are also possible and supported.

An optical chain, in various embodiments, includes a first lens and animage sensor. Additional lenses and/or one or more optical filters maybe included between the first lens of an optical chain module and theimage sensor depending on the particular embodiment. In some cases theremay be one or more optical filters before the first lens.

The use of multiple optical chain modules is well suited for use indevices such as cell phones and/or portable camera devices intended tohave a thin form factor, e.g., thin enough to place in a pocket orpurse. By using multiple optical chains and then combining the capturedimages or portions of the captured images to produce a combined image,improved images are produced as compared to the case where a singleoptical chain module of the same size is used.

While in various embodiments separate image sensors are used for each ofthe individual optical chain modules, in some embodiments the imagesensor of an individual optical chain module is a portion of a CCD orother optical sensor dedicated to the individual optical chain modulewith different portions of the same sensor serving as the image sensorsof different optical chain modules.

In various embodiments, images of a scene area are captured by differentoptical chain modules and then subsequently combined either by theprocessor included in the camera device which captured the images or byanother device, e.g., a personal or other computer which processes theimages captured by the multiple optical chains after offloading from thecamera device which captured the images.

The combined image has, in some embodiments a dynamic range that islarger than the dynamic range of an individual image used to generatethe combined image.

In some such embodiments the sensors of multiple optical chains aremounted on a flat printed circuit board or backplane device. The printedcircuit board, e.g. backplane, can be mounted or coupled to horizontalor vertical actuators which can be moved in response to detected cameramotion, e.g., as part of a shake compensation process which will bediscussed further below. In some such embodiments, pairs of lightdiverting devices, e.g., mirrors, are used to direct the light so thatat least a portion of each optical chain extends perpendicular orgenerally perpendicular to the input and/or sensor plane. Suchembodiments allow for relatively long optical paths which take advantageof the width of the camera by using mirrors or other light divertingdevices to alter the path of light passing through an optical chain sothat at least a portion of the light path extends in a directionperpendicular or generally perpendicular to the front of the cameradevice. The use of mirrors or other light diverting devices allows thesensors to be located on a plane at the rear or front of the cameradevice as will now be discussed in detail.

An exemplary method of zooming video in a continuous manner, inaccordance with some embodiments, includes: providing N optical chains,said N optical chains including at least a first group of optical chainsand a second group of optical chains; discretely transitioning the firstgroup of optical chains from a first fixed focal length to a secondfixed focal length during a first period of time; capturing images fromthe second group of optical chains during said first period of time;discretely transitioning the second group of optical chains from thefirst fixed focal length to the second fixed focal length during asecond period of time; and capturing images from the first group ofoptical chains during said second period of time. Various describedmethods and apparatus use multiple groups of lenses to supportcontinuous zooming with a combination of digital zoom and discrete lensfocal length changes.

An exemplary method of generating video from a sequence of image datacaptured by a camera moving along a path, in accordance with someembodiments, includes: detecting motion, e.g. tracking the path ofmotion, of the moving camera, said moving camera including multipleoptical chains or being a light field camera (Lytro camera), said movingcamera supporting image synthesis from any of a plurality of points ofview within a synthetic aperture region (e.g., set of all the points ofview from which an image can be synthesized by the camera) of saidcamera; and performing a track stabilization operation. In some suchembodiments, the track stabilization operation includes: determining asequence of points of view to be used for synthesizing a sequence ofimages of said video based on said path of motion; and

synthesizing said sequence of images, said synthesized sequence ofimages based on said determined sequence of points of view. In variousembodiments, the exemplary method further includes outputting saidsynthesized sequence of images as said video.

An exemplary camera system, in accordance with some embodiments,includes:

an image capture device including a plurality of camera modules or alight field camera configured to capture a sequence of image data as theimage capture device is moved along a path; a module configured to trackthe path of image capture device; and a track stabilization apparatus.In some such embodiments, said track stabilization apparatus includes amodule configured to determine a sequence of points of view to be usedfor synthesizing a sequence of images of said video based on said pathof motion; and a synthesization module configure to synthesize saidsequence of images, said synthesized sequence of images being based onsaid determined sequence of points of view. In some such embodiments,the camera system further includes an output module configured to outputsaid synthesized sequence of images as said video.

An exemplary method of generating images, in accordance with someembodiments, comprises: detecting an amount of motion, said detectedamount of motion being a detected amount of motion of a camera deviceincluding multiple optical chains or a detected amount of motion betweenan image corresponding to a second time period and an imagecorresponding to a first time period; producing a second referenceimage, from a first plurality of images captured by different opticalchains of said camera device during the second time period; and usingthe second reference image and at least one other image in said firstplurality of images to generate a composite image corresponding to saidsecond time period. In some such embodiments, producing a secondreference image includes at least one of: i) selecting a reference imagefrom images captured by different optical chain modules during saidsecond time period based on the detected amount of motion or ii)synthesizing a reference image from at least two of said multiple imagescaptured by different optical chain modules during said second timeperiod based on the detected amount of motion.

An exemplary camera device, in accordance with some embodiments,includes: a plurality of optical chain modules; a module configured todetect an amount of motion; a module configured to produce a secondreference image, from a first plurality of images (e.g., frames—one peroptical chain) captured by different optical chains of said cameradevice during the second time period, producing a second reference imageincluding at least one of: i) selecting a reference image from imagescaptured by different optical chain modules during said second timeperiod based on the detected amount of motion or ii) synthesizing areference image from at least two of said multiple images captured bydifferent optical chain modules during said second time period based onthe detected amount of motion; and a module configured to use the secondreference image and at least one other image in said first plurality ofimages to generate a composite image corresponding to said second timeperiod.

Numerous additional features and embodiments are described in thedetailed description which follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an exemplary block diagram of an exemplary apparatus, e.g.,camera device, implemented in accordance with one embodiment of thepresent invention.

FIG. 1B illustrates a frontal view of an apparatus implemented inaccordance with an exemplary embodiment of the present invention whichincorporates multiple optical chain modules in accordance with thepresent invention with lenses which are viewable from the front of thecamera.

FIG. 1C, which is a side view of the exemplary apparatus of FIG. 1B,illustrates further details of the exemplary apparatus.

FIG. 1D illustrates a plurality of optical chain modules that can beused in an exemplary device implemented in accordance with theinvention.

FIG. 2 illustrates a camera device implemented in accordance with oneembodiment of the present invention.

FIG. 3A shows an exemplary lens configuration which may be used for theset of outer lenses of the camera device shown in FIGS. 1A-1C.

FIG. 3B illustrates an exemplary filter arrangement which is used in thecamera of FIGS. 1A-1C in some embodiments.

FIG. 3C shows an exemplary inner lens configuration which may, and insome embodiments is, used for a set of inner lenses of the camera deviceshown in FIGS. 1A-1C.

FIG. 4 illustrates an exemplary camera device in which the sets of outerlenses, filters, and inner lenses are mounted on corresponding platters.

FIG. 5 illustrates an exemplary method of producing at least one imageof a first scene area by operating a plurality of optical chain modulesin accordance with one embodiment of the present invention.

FIG. 6 illustrates a computer system which can be used for postprocessing of images captured using a camera device.

FIG. 7 illustrates a frontal view of the outer lens assembly of anapparatus implemented in accordance with one embodiment of the presentinvention where the apparatus incorporates multiple optical chainmodules and outer lenses configured with little or no gaps between thelenses.

FIG. 8 illustrates a frontal view of the outer lenses of a lens assemblyimplemented in accordance with one embodiment of the present inventionwhere the apparatus incorporates multiple optical chain modules withlenses configured with little or no gaps between the lenses butnon-uniform spacing between the optical centers of at least some of thelenses.

FIG. 9 illustrates a camera device including a plurality of opticalchain modules which includes mirrors or another device for changing theangle of light entering the optical chain module and thereby allowing atleast a portion of the optical chain module to extend in a direction,e.g., a perpendicular direction, which is not a straight front to backdirection with respect to the camera device.

FIG. 10 illustrates another camera device including a plurality ofoptical chain modules which includes mirrors or another device forchanging the angle of light entering the optical chain module andthereby allowing at least a portion of the optical chain module toextend in a direction, e.g., a perpendicular direction, which is not astraight front to back direction with respect to the camera device.

FIG. 11 illustrates an additional exemplary camera device in whichmirrors and/or other light redirecting elements are used to alter thepath of light in the optical chains so that both the input lenses and/oropenings through which light enters the optical chains can be arrangedin a plane, and also so that the optical sensors of the optical chainscan be arranged in a plane, while allowing at least a portion of thelight path through the optical chains to extend in a directionperpendicular to the input and/or output planes.

FIG. 12 illustrates an additional exemplary camera device in whichmirrors and/or other light redirecting elements are used to alter thepath of light in the optical chains so that the input lenses and/oropenings, as well as the light sensors of the different optical chains,can be arranged in one or more planes at the front of the camera.

FIG. 13 is a flowchart of an exemplary method of generating video from asequence of image data captured by a camera moving along a path inaccordance with an exemplary embodiment.

FIG. 14 is an exemplary assembly of modules, which may be included in anexemplary device, e.g., a camera device including multiple opticalchains or a light field camera, or an exemplary combination of devices,e.g., a camera device including multiple optical chains or a light fieldcamera, and a computer device external to the camera device,implementing the method of the flowchart of FIG. 13.

FIG. 15 is an exemplary block diagram of an exemplary apparatus, e.g.,camera device, implemented in accordance with an exemplary embodiment ofthe present invention.

FIG. 16 illustrates an exemplary handheld camera, in accordance with anexemplary embodiment of the present invention, with multiple opticalchains, with gyroscopes and with accelerometers, being moved.

FIG. 17A is a first part of a flowchart of an exemplary method ofgenerating images in accordance with an exemplary embodiment.

FIG. 17B is a second part of a flowchart of an exemplary method ofgenerating images in accordance with an exemplary embodiment.

FIG. 17C is a third part of a flowchart of an exemplary method ofgenerating images in accordance with an exemplary embodiment.

FIG. 17D is a fourth part of a flowchart of an exemplary method ofgenerating images in accordance with an exemplary embodiment.

FIG. 18A is a first portion of an assembly of modules which may be usedin accordance with one or more exemplary embodiments.

FIG. 18B is a second portion of the assembly of modules which may beused in accordance with one or more exemplary embodiments.

FIG. 18C is a third portion of the assembly of modules which may be usedin accordance with one or more exemplary embodiments.

FIG. 18D is a fourth portion of the assembly of modules which may beused in accordance with one or more exemplary embodiments.

DETAILED DESCRIPTION

FIG. 1A illustrates an exemplary apparatus 100, sometimes referred tohereinafter as a camera device, implemented in accordance with oneexemplary embodiment of the present invention. The camera device 100, insome embodiments, is a portable device, e.g., a cell phone or tabletincluding a camera assembly. In other embodiments, it is fixed devicesuch as a wall mounted camera.

FIG. 1A illustrates the camera device 100 in block diagram form showingthe connections between various elements of the apparatus 100. Theexemplary camera device 100 includes a display device 102, an inputdevice 106, memory 108, a processor 110, a transceiver interface 114,e.g., a cellular interface, a WIFI interface, or a USB interface, an I/Ointerface 112, and a bus 116 which are mounted in a housing representedby the rectangular box touched by the line leading to reference number100. The input device 106 may be, and in some embodiments is, e.g.,keypad, touch screen, or similar device that may be used for inputtinginformation, data and for instructions. The display device 102 may be,and in some embodiments is, a touch screen, used to display images,video, information regarding the configuration of the camera device,and/or status of data processing being performed on the camera device.In the case where the display device 102 is a touch screen, the displaydevice 102 serves as an additional input device and/or as an alternativeto the separate input device, e.g., buttons, 106. The I/O interface 112couples the display 102 and input device 106 to the bus 116 andinterfaces between the display 102, input device 106 and the otherelements of the camera which can communicate and interact via the bus116. In addition to being coupled to the I/O interface 112, the bus 116is coupled to the memory 108, processor 110, an optional autofocuscontroller 132, a transceiver interface 114, and a plurality of opticalchain modules 130, e.g., N optical chain modules. In some embodiments Nis an integer greater than 2, e.g., 3, 4, 7 or a larger value dependingon the particular embodiment. Images captured by individual opticalchain modules in the plurality of optical chain modules 130 can bestored in memory 108, e.g., as part of the data/information 120 andprocessed by the processor 110, e.g., to generate one or more compositeimages. Multiple captured images and/or composite images may beprocessed to form video, e.g., a series of images corresponding to aperiod of time. Transceiver interface 114 couples the internalcomponents of the camera device 100 to an external network, e.g., theInternet, and/or one or more other devices e.g., memory or stand alonecomputer. Via interface 114 the camera device 100 can and does outputdata, e.g., captured images, generated composite images, and/orgenerated video. The output may be to a network or to another externaldevice for processing, storage and/or to be shared. The captured imagedata, generated composite images and/or video can be provided as inputdata to another device for further processing and/or sent for storage,e.g., in external memory, an external device or in a network.

The transceiver interface 114 of the camera device 100 may be, and insome instances is, coupled to a computer so that image data may beprocessed on the external computer. In some embodiments the externalcomputer has a higher computational processing capability than thecamera device 100 which allows for more computationally complex imageprocessing of the image data outputted to occur on the externalcomputer. The transceiver interface 114 also allows data, informationand instructions to be supplied to the camera device 100 from one ormore networks and/or other external devices such as a computer or memoryfor storage and/or processing on the camera device 100. For example,background images may be supplied to the camera device to be combined bythe camera processor 110 with one or more images captured by the cameradevice 100. Instructions and/or data updates can be loaded onto thecamera via interface 114 and stored in memory 108.

The camera device 100 may include, and in some embodiments does include,an autofocus controller 132 and/or autofocus drive assembly 134. Theautofocus controller 132 is present in at least some autofocusembodiments but would be omitted in fixed focus embodiments. Theautofocus controller 132 controls adjustment of at least one lensposition in the optical chain modules used to achieve a desired, e.g.,user indicated, focus. In the case where individual drive assemblies areincluded in each optical chain module, the autofocus controller 132 maydrive the autofocus drive of various optical chain modules to focus onthe same target. As will be discussed further below, in some embodimentslenses for multiple optical chain modules are mounted on a singleplatter which may be moved allowing all the lenses on the platter to bemoved by adjusting the position of the lens platter. In some suchembodiments the autofocus drive assembly 134 is included as an elementthat is external to the individual optical chain modules with the driveassembly 134 driving the platter including the lenses for multipleoptical chains under control of the autofocus controller 132. While theoptical chain modules will in many embodiments be focused together tofocus on an object at a particular distance from the camera device 100,it is possible for different optical chain modules to be focused todifferent distances and in some embodiments different focus points areintentionally used for different optical chains to increase the postprocessing options which are available.

The processor 110 controls operation of the camera device 100 to controlthe elements of the camera device 100 to implement the steps of themethods described herein. The processor may be a dedicated processorthat is preconfigured to implement the methods. However, in manyembodiments the processor 110 operates under direction of softwaremodules and/or routines stored in the memory 108 which includeinstructions that, when executed, cause the processor to control thecamera device 100 to implement one, more or all of the methods describedherein. Memory 108 includes an assembly of modules 118 wherein one ormore modules include one or more software routines, e.g., machineexecutable instructions, for implementing the image capture and/or imagedata processing methods of the present invention. Individual stepsand/or lines of code in the modules of 118 when executed by theprocessor 110 control the processor 110 to perform steps of the methodof the invention. When executed by processor 110, the data processingmodules 118 cause at least some data to be processed by the processor110 in accordance with the method of the present invention. Theresulting data and information (e.g., captured images of a scene,combined images of a scene, etc.) are stored in data memory 120 forfuture use, additional processing, and/or output, e.g., to displaydevice 102 for display or to another device for transmission, processingand/or display. The memory 108 includes different types of memory forexample, Random Access Memory (RAM) in which the assembly of modules 118and data/information 120 may be, and in some embodiments are stored forfuture use. Read only Memory (ROM) in which the assembly of modules 118may be stored for power failures. Non-volatile memory such as flashmemory for storage of data, information and instructions may also beused to implement memory 108. Memory cards may be added to the device toprovide additional memory for storing data (e.g., images and video)and/or instructions such as programming. Accordingly, memory 108 may beimplemented using any of a wide variety of non-transitory computer ormachine readable mediums which serve as storage devices.

Having described the general components of the camera device 100 withreference to FIG. 1A, various features relating to the plurality ofoptical chain modules 130 will now be discussed with reference to FIGS.1B and 10 which show the camera device 100 from front and sideperspectives, respectively. Dashed line 101 of FIG. 1B indicates a crosssection line corresponding to the FIG. 10 view.

Box 117 represents a key and indicates that OCM=optical chain module andeach L1 represents an outermost lens in an optical chain module. Box 119represents a key and indicates that S=sensor, F=filter, L=lens, L1represents an outermost lens in an optical chain module, and L2represents an inner lens in an optical chain module.

FIG. 1B shows the front of the camera device 100. Rays of light 131,which is light toward the front of the camera assembly, shown in FIG. 10may enter the lenses located in the front of the camera housing. Fromthe front of camera device 100, the camera device 100 appears as arelatively flat device with the outer rectangle representing the camerahousing and the square towards the center of the camera representing theportion of the front camera body in which the plurality of optical chainmodules 130 is mounted.

FIG. 1C, which shows a side perspective of camera device 100,illustrates three of the seven optical chain modules (OCM 1 121, OCM 7145, OCM 4 133) of the set of optical chain modules 130, display 102 andprocessor 110. OCM 1 121 includes an outer lens L1 103, a filter 123, aninner lens L2 125, and a sensor 127. OCM 1 121 further includesautofocus drive (AFD) 129 for controlling the position of lens L2 125,and exposure control device (ECD) 131 for controlling sensor 127. TheAFD 129 includes a motor or other drive mechanism which can move thelens (or sensor) to which it is connected. While the AFD 129 is showncoupled, e.g., connected, to the lens L2 125 and thus can move theposition of the lens L2 as part of a focus operation, in otherembodiments the AFD 129 is coupled to the sensor 127 and moves theposition of the sensor 127, e.g., to change the distance between thesensor 127 and the lens 125 as part of a focus operation. OCM 7 145includes an outer lens L1 115, a filter 147, an inner lens L2 149, and asensor 151. OCM 7 145 further includes AFD 153 for controlling theposition of lens L2 149 and ECD 155 for controlling sensor 151.

OCM 4 133 includes an outer lens L1 109, a filter 135, an inner lens L2137, and a sensor 139. The AFD 153 includes a motor or other drivemechanism which can move the lens (or sensor) to which it is connected.While the AFD 153 is shown coupled, e.g., connected, to the lens L2 149and thus can move the position of the lens L2 as part of a focusoperation, in other embodiments the AFD 149 is coupled to the sensor 151and moves the position of the sensor 151, e.g., to change the distancebetween the sensor 151 and the lens 149 as part of a focus operation.

OCM 4 133 further includes AFD 141 for controlling the position of lensL2 137 and ECD 143 for controlling sensor 139. The AFD 141 includes amotor or other drive mechanism which can move the lens (or sensor) towhich it is connected. While the AFD 141 is shown coupled, e.g.,connected, to the lens L2 137 and thus can move the position of the lensL2 as part of a focus operation, in other embodiments the AFD 141 iscoupled to the sensor 139 and moves the position of the sensor 139,e.g., to change the distance between the sensor 139 and the lens 137 aspart of a focus operation.

While only three of the OCMs are shown in FIG. 1C it should beappreciated that the other OCMS of the camera device 100 may, and insome embodiments do, have the same or similar structure.

FIG. 1C and the optical chain modules (OCMs), also sometimes referred toas optical camera modules, illustrated therein are illustrative of thegeneral structure of OCMs used in various embodiments. However, as willbe discussed in detail below, numerous modifications and particularconfigurations are possible. Many of the particular configurations willbe discussed below with use of reference to the optical camera modulesshown in FIG. 1C. While reference to elements of FIG. 1C may be made, itis to be understood that the OCMs in a particular embodiment will beconfigured as described with regard to the particular embodiment. Thus,for example, the filter may be of a particular color. Similarly, inembodiments where the filter is expressly omitted and described as beingomitted or an element which allows all light to pass, while referencemay be made to the OCMs of FIG. 1C, it should be appreciated that thefilter will be omitted in an embodiment where it is indicated to beomitted or of such a nature that it passes a broad spectrum of light topass if the embodiment is indicated to have a broadband filter. As willbe discussed below, the elements of the different OCMs may, but need notbe, mounted on a common support device, e.g., disc or platter, allowinga set of filters, lenses or sensors of the different optical chains tobe moved as a set. While in the OCMs of FIG. 1C mirrors are not shown,as will be discussed below, in at least some embodiments one or moremirrors are added to the OCMs to all light to be directed, e.g., toincrease the length of the optical path or make for a more convenientinternal component configuration. It should be appreciated that each ofthe OCMS 121, 145, 133, shown in FIG. 1C will have their own opticalaxis which corresponds to the path light entering the particular OCMwill follow as it passes from the lens 103, 115, or 109 at the front ofthe optical chain and passes through the OCM to the corresponding sensor127, 151, 139.

While the processor 110 is not shown being coupled to the AFD, ECD andsensors 127, 151, 139 it is to be appreciated that such connectionsexist and are omitted from FIG. 10 to facilitate the illustration of theconfiguration of the exemplary OCMs.

As should be appreciated the number and arrangement of lens, filtersand/or mirrors can vary depending on the particular embodiment and thearrangement shown in FIG. 1C is intended to be exemplary and tofacilitate an understanding of the invention rather than limiting innature.

The front of the plurality of optical chain modules 130 is visible inFIG. 1B with the outermost lens of each optical chain module appearingas a circle represented using a solid line (OCM 1 L1 103, OCM 2 L1 105,OCM 3 L1 107, OCM 4 L1 109, OCM 5 L1 111, OCM 6 L1 113, OCM 7 L1 115).In the FIG. 1B example, the plurality of optical chain modules 130include seven optical chain modules, OCM 1 121, OCM 2 157, OCM 3 159,OCM 4 133, OCM 5 171, OCM 6 173, OCM 7 145, which include lenses (OCM 1L1 103, OCM 2 L1 105, OCM 3 L1 107, OCM 4 L1 109, OCM 5 L1 111, OCM 6 L1113, OCM 7 L1 115), respectively, represented by the solid circles shownin FIG. 1B. The lenses of the optical chain modules are arranged to forma pattern which is generally circular in the FIG. 1B example when viewedas a unit from the front. While a circular arrangement is preferred insome embodiments, non-circular arrangements are used and preferred inother embodiments. In some embodiments while the overall pattern isgenerally or roughly circular, different distances to the center of thegeneral circle and/or different distances from one lens to another isintentionally used to facilitate generation of a depth map and blockprocessing of images which may include periodic structures such asrepeating patterns without the need to identify edges of the repeatingpattern. Such repeating patterns may be found in a grill or a screen.

Note that the individual outer lenses, in combination, occupy an areathat might otherwise have been occupied by a single large lens. Thus,the overall total light capture area corresponding to the multiplelenses of the plurality of chain modules OCM 1 to OCM 7, also sometimesreferred to as optical camera modules, approximates that of a lenshaving a much larger opening but without requiring a single lens havingthe thickness which would normally be necessitated by the curvature of asingle lens occupying the area which the lenses shown in FIG. 1B occupy.

While gaps are shown between the lens openings of the optical chainmodules OCM 1 to OCM 7, it should be appreciated that the lenses may bemade, and in some embodiments are, made so that they closely fittogether minimizing gaps between the lenses represented by the circlesformed by solid lines. While seven optical chain modules are shown inFIG. 1B, it should be appreciated that other numbers of optical chainmodules are possible.

As will be discussed below, the use of seven optical chain modulesprovides a wide degree of flexibility in terms of the types of filtercombinations and exposure times that can be used for different colorswhile still providing an optical camera module that can be used toprovide an image for purposes of user preview of the image area andselection of a desired focal distance, e.g., by selecting an object inthe preview image which is to be the object where the camera modules areto be focused.

For example, in some embodiments, at least some of the different opticalchain modules include filters corresponding to a single color therebyallowing capture of a single color at the full resolution of the imagesensor, e.g., the sensor does not include a Bayer filter. In oneembodiment two optical chain modules are dedicated to capturing redlight, two optical chain modules are dedicated to capturing green lightand two optical chain modules are dedicated to capturing blue light. Thecenter optical chain module may include a RGB filter or opening whichpasses all colors with different portions of the sensor of the centeroptical chain module being covered by different color filters, e.g., aBayer pattern with the optical chain module being used to capture allthree colors making it easy to generate color preview images withouthaving to process the output of multiple optical chain modules togenerate a preview image.

The use of multiple optical chains such as shown in the FIG. 1A-1Cembodiment has several advantages over the use of a single opticalchain.

Using multiple optical chains allows for noise averaging. For example,given the small sensor size there is a random probability that oneoptical chain may detect a different number, e.g., one or more, photonsthan another optical chain. This may represent noise as opposed toactual human perceivable variations in the image being sensed. Byaveraging the sensed pixel values corresponding to a portion of animage, sensed by different optical chains, the random noise may beaveraged resulting in a more accurate and pleasing representation of animage or scene than if the output of a single optical chain was used.

As should be appreciated, different wavelengths of light will be bent bydifferent amounts by the same lens. This is because the refractive indexof glass (or plastic) which the lens is made of changes with wavelength.Dedication of individual optical chains to a particular color allows forthe lenses for those optical chains to be designed taking intoconsideration the refractive index of the specific range of wavelengthfor that color of light. This can reduce chromatic aberration andsimplify lens design. Having multiple optical chains per color also hasthe advantage of allowing for different exposure times for differentoptical chains corresponding to a different color. Thus, as will bediscussed further below, a greater dynamic range in terms of lightintensity can be covered by having different optical chains usedifferent exposure times and then combining the result to form thecomposite image, e.g., by weighting the pixel values output by thesensors of different optical chains as a function of exposure time whencombing the sensed pixel values to generate a composite pixel value foruse in a composite image. Given the small size of the optical sensors(pixels) the dynamic range, in terms of light sensitivity, is limitedwith the sensors becoming easily saturated under bright conditions. Byusing multiple optical chains corresponding to different exposure timesthe dark areas can be sensed by the sensor corresponding to the longerexposure time while the light areas of a scene can be sensed by theoptical chain with the shorter exposure time without getting saturated.Pixel sensors of the optical chains that become saturated as indicatedby a pixel value indicative of sensor saturation can be ignored, and thepixel value from the other, e.g., less exposed, optical chain can beused without contribution from the saturated pixel sensor of the otheroptical chain. Weighting and combining of non-saturated pixel values asa function of exposure time is used in some embodiments. By combiningthe output of sensors with different exposure times a greater dynamicrange can be covered than would be possible using a single sensor andexposure time.

FIG. 1C is a cross section perspective of the camera device 100 shown inFIGS. 1A and 1B. Dashed line 101 in FIG. 1B shows the location withinthe camera device to which the cross section of FIG. 1C corresponds.From the side cross section, the components of the first, seventh andfourth optical chains are visible.

As illustrated in FIG. 1C despite including multiple optical chains thecamera device 100 can be implemented as a relatively thin device, e.g.,a device less than 2, 3 or 4 centimeters in thickness in at least someembodiments. Thicker devices are also possible, for example devices withtelephoto lenses and are within the scope of the invention, but thethinner versions are particularly well suited for cell phones and/ortablet implementations.

As illustrated in the FIG. 1C diagram, the display device 102 may beplaced behind the plurality of optical chain modules 130 with theprocessor 110, memory and other components being positioned, at least insome embodiments, above or below the display and/or optical chainmodules 130. As will be discussed below, and as shown in FIG. 10, eachof the optical chains OCM 1 121, OCM 7 145, OCM 4 133 may, and in someembodiments do, include an outer lens L1, an optional filter F, and asecond optional lens L2 which proceed a sensor S which captures andmeasures the intensity of light which passes through the lens L1, filterF and second lens L2 to reach the sensor S. The filter may be a colorfilter or one of a variety of other types of light filters.

In FIG. 1C, each optical chain module includes an auto focus drive (AFD)also sometimes referred to as an auto focus device which can alter theposition of the second lens L2, e.g., move it forward or back, as partof a focus operation. An exposure control device (ECD) which controlsthe light exposure time of the sensor to which the ECD corresponds, isalso included in each of the OCMs shown in the FIG. 1C embodiment. TheAFD of each optical chain module operates under the control of theautofocus controller 132 which is responsive to user input whichidentifies the focus distance, e.g., by the user highlighting an objectin a preview image to which the focus is to be set. The autofocuscontroller while shown as a separate element of the device 100 can beimplemented as a module stored in memory and executed by processor 110.

Note that while supporting a relatively large light capture area andoffering a large amount of flexibility in terms of color filtering andexposure time, the camera device 100 shown in FIG. 1C is relatively thinwith a thickness that is much less, e.g., ⅕th, 1/10th, 1/20th or evenless than the overall side to side length or even top to bottom lengthof the camera device visible in FIG. 1B.

FIG. 1D illustrates a plurality of optical chain modules 160 that can beused in an exemplary device implemented in accordance with theinvention. The optical chain modules (OCMs) shown in FIG. 1D areillustrative of the general structure of OCMs used in variousembodiments. However, as will be discussed in detail below, numerousmodifications and particular configurations are possible. Many of theparticular configurations will be discussed below with use of referenceto the optical camera modules shown in FIG. 1D to support the particularexemplary embodiments. While reference to elements of FIG. 1D may andwill be made with regard to particular embodiments, it is to beunderstood that the OCMs in a particular embodiment will be configuredas described with regard to the particular embodiment. Thus, forexample, in a particular embodiment one or of the OCMS may use filtersof a particular color or may even omit the filter 164, 164′. 164″ or164′″. Similarly, in embodiments where the filter is expressly omittedand described as being omitted or an element which allows all light topass, while reference may be made to the OCMs of FIG. 1D, it should beappreciated that the filter will be omitted in such an embodiment whereit is expressly indicated to be omitted or of such a nature that itpasses a broad spectrum of light to pass if the embodiment is indicatedto have a broadband filter. As will be discussed below, the elements ofthe different OCMs may, but need not be, mounted on a common supportdevice, e.g., disc or platter, allowing a set of filters, lenses orsensors of the different optical chains to be moved as a set. While inthe OCMs of FIG. 1D mirrors are not shown, as will be discussed below,in at least some embodiments one or more mirrors are added to the OCMsto all light to be directed, e.g., to increase the length of the opticalpath or make for a more convenient internal component configuration. Itshould be appreciated that each of the OCMS 164, 164′, 164″. 164′, shownin FIG. 10 will have their own optical axis which corresponds to thepath light entering the particular OCM will follow as it passes from thelens 162, 162′. 162″, 162′″ at the front of the optical chain and passesthrough the OCM to the corresponding sensor 168, 168′, 168″, 168′.

The plurality of optical chain modules 160 includes N exemplary opticalchain modules as illustrated in FIG. 1D where N may be any number butusually a number greater than one, and in many cases greater than 2, 6or even a larger number. The plurality of optical chain modules 160includes a first optical chain module (OCM) 161, a second optical chainmodule 161′, a third optical chain module 161″, . . . , and Nth opticalchain module 161′″.

Each optical chain module illustrated in FIG. 1D includes many or all ofthe same elements shown in each optical chain illustrated in FIG. 1Csuch as, e.g., optical chain module 121. The first exemplary OCM 161includes an outer lens 162, a filter 164, an inner lens 166, a sensor168, an auto focus drive (AFD) 169 and an exposure control device (ECD)170. Each of the other optical chain modules include similar elements asdescribed above with regard to the first OCM 160, with the like elementsin each of the other optical chain modules being identified using aprime (′), double prime (″), or triple prime (′″). For example, theexemplary second OCM 161′ includes an outer lens 162′, a filter 164′, aninner lens 166′, a sensor 168′, an auto focus drive (AFD) 169′ and anexposure control device (ECD) 170′, the exemplary third OCM 161″includes an outer lens 162″, a filter 164″, an inner lens 166″, a sensor168″, an auto focus drive (AFD) 169″ and an exposure control device(ECD) 170″ and so on. Similarly, the Nth OCM 161′ includes an outer lens162′″, a filter 164′″, an inner lens 166′″, a sensor 168′″, an autofocus drive (AFD) 169′″ and an exposure control device (ECD) 170′″. Theoperation and functionality of each of the OCMs and their elements isthe same as or similar the functionality of optical chain modulesdiscussed earlier with respect to FIG. 1C and thus will not be repeated.Note that two versions of the AFD 169, 169′, 169″ or 169′″ are shown ineach optical chain module with the AFD connected to a lens being shownusing solid lines and an alternative AFD shown using dashed lines beingconnected to the sensor 168, 168′, 168″ or 168′″. The AFD shown withdashed lines adjusts the position of the sensor 168, 168′, 168″ or 168′″to which it is connected as part of an autofocus operation, e.g., movingthe sensor forward or backward to alter distance between the sensor anda lens. The AFD shown in solid lines is used in systems where a lensrather than a sensor is moved as part of an AFD operation. In someembodiments the AFD controls the position of a lens and/or sensor inwhich case the AFD module is connected to both a lens support mechanismor lens and the sensor.

The plurality of optical chain modules 160 of FIG. 1D can be used as,e.g., the plurality of optical modules 130 of the exemplary device 100or any other device implemented in accordance with the invention. Thenumber and particular configuration of optical chains in the step ofoptical chains 160 maybe as per various embodiments which will bedescribed in the following detailed description. Accordingly, while aparticular embodiment may be described in one more subsequent portionsof this application, it is to be understood that the optical chains 160may be used in such embodiments with the particular configuration offilters, lens, and element supports being as described with respect tothe particular exemplary embodiment being discussed.

FIG. 2 illustrates a camera device 200 implemented in accordance withthe invention. The FIG. 2 camera device 200 includes many or all of thesame elements shown in the device 100 of FIGS. 1A-1C. Exemplary cameradevice 200 includes a plurality of optical chain modules (OCM 1 205, OCM2 207, . . . , OCM N 209, a processor 211, memory 213 and a display 215,coupled together. OCM 1 205 includes outer lens L1 251, filter 253,inner lens L2 255, sensor 1 257, AFD 259 and ECD 261. In someembodiments, processor 211 of camera device 200 of FIG. 2 is the same asprocessor 110 of device 100 of FIG. 1A, memory 213 of device 200 of FIG.2 is the same as memory 108 of device 100 of FIG. 1A, and display 215 ofdevice 200 of FIG. 2 is the same as display 102 of device 100 of FIG.1A.

OCM 2 207 includes outer lens L1 263, filter 265, inner lens L2 267,sensor 2 269, AFD 271 and ECD 273. OCM N 209 includes outer lens L1 275,filter 277, inner lens L2 279, sensor N 281, AFD 283 and ECD 285. Box217, which represents a key, indicates that ECD=exposure control deviceand AFD=auto focus drive.

In the FIG. 2 embodiment the optical chain modules (optical chain module1 205, optical chain module 2 207, . . . , optical chain module N 209)are shown as independent assemblies with the autofocus drive of eachmodule being a separate AFD element (AFD 259, AFD 271, AFD 283),respectively.

In FIG. 2, the structural relationship between the various lenses andfilters which precede the sensor in each optical chain module can beseen more clearly. While three elements, e.g. two lenses (see columns201 and 203 corresponding to L1 and L2, respectively) and the filter(corresponding to column 202) are shown in FIG. 2 before each sensor, itshould be appreciated that a much larger combination of lenses and/orfilters may precede the sensor of one or more optical chain modules withanywhere from 2-10 elements being common and an even larger number ofelements being used in some embodiments, e.g., high end embodimentsand/or embodiments supporting a large number of filter and/or lensoptions.

In some but not all embodiments, optical chain modules are mounted inthe camera device to extend from the front of the camera device towardsthe back, e.g., with multiple optical chain modules being arranged inparallel. Filters and/or lenses corresponding to different optical chainmodules may, and in some embodiments are, arranged in planes extendingperpendicular to the front to back direction of the camera device fromthe bottom of the camera device towards the top of the camera device.While such a mounting arrangement is used in some embodiments, otherarrangements where the optical chain modules are arranged at differentangles to one another and/or the camera body are possible.

Note that the lenses/filters are arranged in planes or columns in thevertical dimension of the camera device 200 to which reference numbers201, 202, 203 correspond. The fact that the lenses/filters are alignedalong vertical planes allows for a manufacturing and structuralsimplification that is used in some embodiments. That is, in someembodiments, the lenses and/or filters corresponding to a plane 201,202, 203 are formed or mounted on a platter or plate. The term platterwill be used for discussion purposes but is not intended to be limiting.The platter may take the form of a disc but non-round platters are alsocontemplated and are well suited for some embodiments. In the case ofplastic lenses, the lenses and platter may be molded out of the samematerial in a single molding operation greatly reducing costs ascompared to the need to manufacture and mount separate lenses. As willbe discussed further, platter based embodiments allow for relativelysimple synchronized focus operations in that a platter may be movedfront or back to focus multiple OCMs at the same time. In addition, aswill be explained, platters may be moved or rotated, e.g., along acentral or non-central axis, to change lenses and or filterscorresponding to multiple optical chain modules in a single operation. Asingle platter may include a combination of lenses and/or filtersallowing, e.g., a lens to be replaced with a filter, a filter to bereplaced with a lens, a filter or lens to be replaced with anunobstructed opening. As should be appreciated the platter basedapproach to lens, filter and/or holes allows for a wide range ofpossible combinations and changes to be made by simple movement of oneor more platters. It should also be appreciated that multiple elementsmay be combined and mounted together on a platter. For example, multiplelenses, filters and/or lens-filter combinations can be assembled andmounted to a platter, e.g., one assembly per optical chain module. Theassemblies mounted on the platter for different optical chains may bemoved together, e.g., by rotating the platter, moving the platterhorizontally or vertically or by moving the platter using somecombination of one or more such movements.

While platters have been described as being moved to change elements inan optical chain, they can, and in some embodiments are, moved for imagestabilization purposes. For example, a platter having one or more lensesmounted thereon can be moved as part of an image stabilizationoperation, e.g., to compensate for camera motion.

While mounting of lenses and filters on platters has been discussed, itshould also be appreciated that the sensors of multiple optical chainscan be mounted on a platter. For example, sensors without color filtersmay be replaced with sensors with color filters, e.g., Bayer patternfilters. In such an embodiment sensors can be swapped or changed whileleaving one or more components of one or more optical chains in place.

Note from a review of FIG. 2 that in some embodiments, e.g., largerfocal length telephoto applications, the elements, e.g., filters/lensescloser to the sensor of the optical chain module, are smaller in sizethan the outer most lenses shown in column 201. As a result of theshrinking size of the lenses/filters, space becomes available betweenthe lenses/filters within the corresponding platter.

FIGS. 3A through 3C provide perspective views of the different planes201, 202, 203 shown in FIG. 2. As shown in FIG. 3A, the outer lenses L1(OCM 1 L1 251, OCM 2 L1 263, OCM 3 L1 264, OCM 4 L1 266, OCM 5 L1 268,OCM 6 L1 270, OCM 7 L1 272) occupy much of the outer circular areacorresponding to the front of the camera modules as previously shown inFIG. 1B. However, as shown in FIG. 3B the filters (OCM 1 F 253, OCM 2 F265, OCM 3 F 274, OCM 4 F 276, OCM 5 F 278, OCM 6 F 280, OCM 7 F 282)corresponding to plane 202 occupy less space than the lenses shown inFIG. 3A while the inner lenses L2 (OCM 1 L2 255, OCM 2 L2 267, OCM 3 L2284, OCM 4 L2 286, OCM 5 L2 288, OCM 6 L2 290, OCM 7 L2 292) shown inFIG. 3C occupy even less space. In some embodiments, where N=7, outerlens L1 275, filter F 277, and inner lens L2 279 of FIG. 2 are the sameas OCM 7 L1 272 of FIG. 3A, OCM 7 F 282 of FIG. 3B and OCM 7 L2 292 ofFIG. 3C, respectively.

The decreasing size of the inner components allow multiple lenses and/orfilters to be incorporated into a platter corresponding to one or moreof the inner planes. Consider for example that an alternative filter F′or hole could be mounted/drilled below or next two each filter F of aplatter corresponding to plan 202 and that by shifting the position orplatter vertically, horizontally or a combination of horizontally andvertically, the filter F can be easily and simply replaced with anotherfilter or hole. Similarly the lenses L2 may be replaced by alternativelenses L2′ by shifting a platter of lenses corresponding to plane 203.In some embodiments, the platter may also be rotated to support changes.The rotation may be an off center rotation and/or may be performed incombination with one or more other platter position changes.

A camera device 60 which includes platters of lenses and/or filters (61,62, 63) is shown in FIG. 4. Camera device 60 includes a plurality ofoptical chain modules (optical chain module 1 69, optical chain module 270, . . . , optical chain module N 71), processor 72, memory 73, anddisplay 74 coupled together via bus 75. Optical chain module 1 69includes sensor 1 79 and ECD 80; optical chain module 2 70 includessensor 2 84 and ECD 85; and optical chain module N 71 includes sensor N89 and ECD 90. In some embodiments, processor 72, memory 73, display 74,and autofocus controller 76 of device 60 of FIG. 4 are the same asprocessor 110, memory 108, display 102, and autofocus controller 132 ofdevice 100 of FIG. 1A.

Element 61 represents a platter of outer lenses L1 with 3 of the lenses(76, 81, 86) being shown as in the FIG. 1C example. Additional lensesmay be, and often are, included on the platter 61 in addition to theones shown. For example, in a seven optical chain module embodiment suchas shown in FIG. 1, platter 61 would include seven outer lenses. Notethat the thickness of the platter 61 need not exceed the maximumthicknesses of the lenses and from a side perspective is much thinnerthan if a single lens having a similar curvature to that of theindividual lenses L1, but with the single lens being larger, occupiedthe same area as all the 7 lenses on the platter 61. Platter 62 includesthe filters F, which include the three filters (77, 82, 87) whileplatter 63 includes the inner lenses L2, which include the three lenses(78, 83, 88). As can be appreciated the camera device 60 is the same asor similar to the camera device of FIG. 1C and FIG. 2 but with thelenses and filters being mounted on platters which may be moved betweenthe front and back of the camera to support autofocus or horizontallyand/or vertically to support lens/filter changes.

Auto focus drive 66 is used to move platter 63 forward or backward aspart of a focus operation, e.g., under control of the autofocuscontroller 76 which may be, and often is, included in the camera device60. A filter shift drive (FSD) 65 is included in embodiments whereshifting of the platter 62 is supported as part of a filter changeoperation. The FSD 65 is responsive to the processor 72 which operatesin response to user selection of a particular mode of operation and/oran automatically selected mode of operation and can move the platter 62vertically, horizontally or in some combination of vertical andhorizontal motion to implement a filter change operation. The FSD 62 maybe implemented with a motor and mechanical linkage to the platter 62. Insome embodiments, the platter 62 may also be rotated to support changes.The rotation may be an off center rotation and/or may be performed incombination with one or more other platter position changes.

A lens shift drive (LSD) 67 is included in embodiments where shifting ofthe platter 63 is supported as part of a filter change operation. TheLSD 67 is responsive to the processor 72 which operates in response touser selection of a particular mode of operation and/or an automaticallyselected mode of operation and can move the platter 63 vertically,horizontally or in some combination of vertical and horizontal motion toimplement a lens change operation. The LSD 67 may be implemented with amotor and mechanical linkage to the platter 63. In some embodiments, theplatter 63 may also be rotated to support changes. The rotation may bean off center rotation and/or may be performed in combination with oneor more other platter position changes.

Method 300 of FIG. 5 illustrates one exemplary method of producing atleast one image of a first scene area in accordance with the presentinvention. The processing steps of the method 300 of FIG. 5 will now beexplained in view of the camera device 100 of FIG. 1A.

The method 300 of FIG. 5 starts at start step 302 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 302 to step 304. In step 304, user input isreceived to control the capture of at least one image of the first scenearea. The user input is received via input device 106 which may be, andin some embodiments is, a button or touch sensitive screen. In optionalsub-step 306, the user input may, and in some embodiments does, indicatea portion of the first scene area that is to be focused, e.g., in animage to be captured or a combined image to be generated from two ormore captured images. From step 304 processing proceeds to step 308.

In step 308, a plurality of three or more optical chain modules (OCMs),e.g., optical chain modules 130 of FIG. 1A, are operated in parallel tocapture images of the first scene area, said images including at least afirst image of said first scene area, a second image of said first scenearea, and a third image of said first scene area. In some embodimentseach one of the first, second and third optical chain modules captures acorresponding one of the first, second and third image respectively. Insome embodiments, operating a plurality of three or more optical chainmodules in parallel to capture images of the first scene area, saidimages including at least a first image of said first scene area, asecond image of said first scene area, and a third image of said firstscene area includes sub-processing steps 310, 312, and 314.

In sub-step 310 a first optical chain module is operated to capture afirst image 316 of the first scene area. In most, but not all,embodiments, on capture of the first image 316, the image data and otherdata such as camera device configuration information associated with thefirst image is stored in the data/information 120 portion of memory 108for later processing, output or display. In parallel with the processingof sub-step 310 processing of sub-steps 312 and 314 also occur. Insub-step 312 a second optical chain module is operated to capture asecond image 318 of the first scene area. In most, but not all,embodiments on capture of the second image 318, the image data and otherdata such as camera device configuration information associated with thesecond image is stored in the data/information 120 portion of memory 108for later processing, output or display. In sub-step 314 a third opticalchain module is operated to capture a third image 320 of the first scenearea. In most, but not all, embodiments on capture of the third image320, the image data and other data such as camera device configurationinformation associated with the third image is stored in thedata/information 120 portion of memory 108 for later processing, outputor display. Processing then proceeds from step 308 to step 322.

In some embodiments, each optical chain module of the plurality ofoptical chain modules includes a lens and the lenses of the plurality ofthe optical chain modules are arranged along a circle. For example, whenthere are three optical chain modules, i.e., a first optical chainmodule, a second optical chain module, and a third optical chain module,the first optical chain module includes a first lens, the second opticalchain module includes a second lens, and the third optical chain moduleincludes a third lens. The first, second and third lenses are arrangeduniformly along a circle, e.g. on the vertices of an equilateraltriangle. In some embodiments the camera device 100 includes a fourthoptical chain module including a fourth lens, said fourth lens beingpositioned in the center of the circle. Each of the first, second, thirdand fourth lens may be, and in some embodiments of the present inventionare, the outer lens of each of their respective optical chain modulesand are all positioned in the same plane. More generally, in someembodiments of the present invention, there are a plurality of N opticalchain modules each including a lens. N−1 lenses of the plurality ofoptical chain modules are arranged along a circle with Nth lens beingpositioned in the center of the circle. FIG. 1B illustrates and exampleof a camera device 100 with seven optical chain modules which include 7outer lenses shown as circles, i.e., OCM1, OCM2, OCM3, OCM4, OCM5, OCM6,and OCM7. The outer lens of optical chain modules OCM 1, OCM2, OCM3,OCM4, OCM5, and OCM6 are arranged along a circle and the outer lens ofoptical chain module OCM7 is positioned in the center of the circle.

In some embodiments of the present invention, the first optical chainmodule includes in addition to the first lens an image sensor referredto as a first image sensor. In some embodiments of the presentinvention, the second optical chain module includes an image sensorreferred to as a second image sensor. In some embodiments of the presentinvention, the third optical chain includes an image sensor referred toas a third image sensor. In some embodiments of the present inventionthe plurality of lenses of the plurality of optical chain modules aremounted in a cell phone housing with the plurality of lenses oriented inthe same direction and in the same plane of the housing. For example inthe case of three optical chain modules, in some embodiments of thepresent invention, the first, second and third lenses of the first,second, and third optical chain modules respectively are mounted in acell phone housing and are oriented in the same direction and in thesame plane of the housing.

In step 322, said first, second, and third images are processed byprocessor 110 to generate a first combined image 326 of said first scenearea. In some embodiments, including those embodiments of the presentinvention in which user input is received indicating a portion of thefirst scene area to be focused in the combined image, step 322 may, andin some embodiments does, include sub-step 324 wherein pixel positionson at least one of said first, second, and third images is shifted priorto generating said first combined image to align the portion of thefirst scene to be focused. Processing then proceeds to step 328 wherethe generated combined image is stored in data/information 120 of memory108, e.g., for potential later display, output from the camera device,and/or additional processing and/or displayed on display 102 of cameradevice 100.

In some embodiments, processing step 322 and/or sub-step 324 areperformed on an external device such as a computer. In such cases, thefirst, second and third images are outputted from the camera device 100via transceiver 114 to the external computer for processing to generatethe first combined image 326. The first combined image may then bestored in memory associated with the external device and/or displayed ona display associated with the external computer. In some embodiments ofthe present invention, the first combined image of the first scene areaincludes the same or fewer pixel values than either of said first,second or third images.

From step 328 processing proceeds to step 304 where processing continuesand the method is repeated.

In some embodiments of the present invention, the size of the diameterof the first, second and third lens of the first, second, and thirdoptical chain modules respectively are the same and the sensors of thefirst, second and third optical chain modules have the same number ofpixels. In other embodiments of the present invention, one or moreoptical chain modules may, and in some embodiments do, have lenses withdifferent diameter sizes and/or sensors with different numbers ofpixels. In some embodiments of the present invention, the first, secondand third lenses of the first, second and third optical chain modulesrespectively, are less than 2 cm in diameter and each of the first,second and third image sensors of the first, second and third opticalchain modules support at least 8 Mpixels. In some embodiments of thepresent invention, the first and second lenses are each less than 2 cmin diameter and each of the first and second image sensors support atleast 5 Mpixels. However in many embodiments the image sensors support 8Mpixels or even more and in some embodiments the lenses are larger than2 cm. Various combinations of lens and sensors may be used with avariety of lens sizes being used for different optical chains in someembodiments. In addition different optical chains may use lenses withdifferent shapes, e.g., while the lens may be a spherical lens theperimeter of the lens may be cut into one of a variety of shapes. In oneembodiment, lenses of different optical chain modules are shaped andarranged to minimize gaps between lenses. Such an approach can have theadvantage of resulting in a smoother blur with regard to portions ofcaptured images which are out of focus when combining images captured bydifferent optical chain modules and result in an overall image whichmore closely approximates what might be expected had a single large lensbeen used to capture the scene shown in the combined image.

In accordance with some aspects of the present invention, the diametersize and arrangement of the lenses of the plurality of optical modulesmay and do vary. Similarly the number of pixels supported by the sensorsof each of the plurality of optical modules may also vary for exampledepending on the desired resolution of the optical chain module.

In some embodiments, different shifts are used for different portions ofthe scene to create a single composite image. In some embodiments, thegenerated combined image is a panoramic image.

In various embodiments, the optical chain modules are independentlyfocused to the same focal distance. In some embodiments, the opticalchain modules are focused together. In some such embodiments, theoptical chain modules are focused together by moving a platter on whichlenses corresponding to different optical chains are mounted.

FIG. 6 illustrates a computer system which can be used for postprocessing of images captured using a camera device. The computer system1400 includes a display 1402, Input/Output (I/O) interface 1412,receiver 1404, input device 1406, transceiver interface 1414, processor1410 and memory 1408. Memory 1408 includes a first portion 1424including data/information 1420 and an assembly of modules 1418, and asecond portion 1426 including storage 1422. The memory 1408 is coupledto the processor 1410, I/O interface 1412 and transceiver interface 1414via bus 1416 through which the elements of the computer system 1400 canexchange data and can communicate with other devices via the I/Ointerface 1412 and/or interface 1414 which can couple the system 1400 toa network and/or camera apparatus. It should be appreciated that viainterface 1414 image data can be loaded on to the computer system 1400and subject to processing, e.g., post capture processing. The images maybe stored in the storage portion 1422 of memory 1408 for processing.Data/information 1420 includes, e.g., intermediate processing data andinformation and criteria used for processing e.g., weightinginformation, exposure time information, etc. The assembly of modules1418 includes one or more modules or routines which, when executed bythe processor 1410, control the computer system to implement one or moreof the image processing operations described in the present application.The output of multiple optical receiver chains can be, and in someembodiments is, combined to generate one or more images. The resultingimages are stored in the storage portion of the memory 1408 prior tobeing output via the network interface 1414, though another interface,or displayed on the display 1402. Thus, via the display 1402 a user canview image data corresponding to one or more individual optical chainmodules as well as the result, e.g., image, generated by combining theimages captured by one or optical chain modules.

FIG. 7 illustrates a frontal view of the outer lenses of an apparatus1605, e.g., a camera device, implemented in accordance with oneembodiment of the present invention which incorporates multiple opticalchain modules and which is designed to have little or no gaps betweenthe outer most lenses of the different optical chain modules. The outermost lenses may be the aperture stop lenses in the FIG. 16 embodiment.Apparatus 1605 of FIG. 7 includes 7 optical chain modules OCM1, OCM2,OCM3, OCM4, OCM5, OCM6 and OCM7 with the outer lens plane correspondingto lenses L1 as viewed from the front of the camera device being shownin FIG. 7.

The 7 optical chain modules are, e.g., optical chain modules (OCM 1 161,OCM 2 161′, OCM 3 161″, . . . , OCM 7 161′″, of FIG. 1D with the outerlens (OCM 1 L1 162, OCM 2 L1 162′, OCM 3 L1 162″, . . . , OCM 7 L1162′″) being outer lenses (OCM 1 L1 1607, OCM 2 L1 1609, OCM 3 L1 1611,. . . , OCM 7 L1 1619) of FIG. 16, respectively.

The outer lenses L1 of optical chain modules 1, 2, 3, 4, 5, and 6, OCM 1L1 1607, OCM 2 L1 1609, OCM 3 L1 1611, OCM 4 L1 1613, OCM 5 L1 1615, OCM6 L1 1617, are positioned so as to surround the outer lens L1 of theoptical chain module 7, OCM 7 L1 1619. The outer lens L1 of the opticalchain module 7 1619 being formed in the shape of a hexagon, i.e., a sixsided polygon. The outer lenses L1 of optical chain modules 1, 2, 3, 4,5 and 6 (1607, 1609, 1611, 1613, 1615, 1617) being of same shape andsize and when combined with lens L1 of optical module 7 (1619) forming acircle. The optical center of each lens L1 of optical chain modules (OCM1 L1 1607, OCM 2 L1 1609, OCM 3 L1 1611, OCM 4 L1 1613, OCM 5 L1 1615,OCM 6 L1 1617) shown as a dark solid dot (1621, 1623, 1625, 1627, 1629,1631) on the dashed circle 1651. The optical center of lens L1 1619 ofoptical chain module 7 shown as a dot 1633 in the center of the hexagonand also in center of the dashed line 1651. A block separator or otherlight block may be used between the lenses to stop light leakage betweenthe different lenses. The dots (1621, 1623, 1625, 1627, 1629, 1631,1633) in FIG. 7 represent the optical center of the individual lenses(1607, 1609, 1611, 1613, 1615, 1617, 1619), respectively. In someembodiments each outermost lens is a round convex lens with itsparameter cut to the shape shown in FIG. 7 so that the lenses fightclosely together. The little or no gap between the front lenses, e.g.,the total area of the gap between the lenses occupies less than 5% ofthe total area of the front area of the lens assembly, e.g., circleshown in FIG. 16, occupied by the lenses when assembled together. Thelack of or small size of the gaps facilitates generating combined imageswith a desirable bokehs or blurs in the combined image with regard toimage portions which are out of focus, e.g., in some cases without theneed for extensive and potentially complex processing to generate thecombined image.

In FIG. 7, circle 1603 represents a circular aperture for the cameradevice 1605. In other embodiments, the aperture for the camera device1605 is a polygon shaped aperture. The plurality of lenses (1607, 1609,1611, 1613, 1615, 1615, 1617, 1619) are configured to partition theaperture 1603 into a plurality of light capture areas (1641, 1643, 1645,1647, 1649, 1650, 1653), occupying substantially the entire area of thefirst aperture.

In some embodiments, the seven optical chains included in camera device1605 are the N optical chains (161, 161′, 161″ . . . , 161′″), whereN=7, where the outer lenses configuration of FIG. 16 is used. Forexample, OCM 1 L1 162 of FIG. 1D is OCM L1 1607 of FIG. 16, OCM 2 L1162′ of FIG. 1D is OCM 2 L1 1609 of FIG. 16, OCM 3 L1 162″ of FIG. 1D isOCM 3 L1 1611 of FIG. 16, . . . . , and OCM N L1 162′″ of FIG. 1D is OCM7 L1 1619 of FIG. 16.

In various embodiments, the sensor included in each optical chain incamera device 1605 is a semiconductor sensor. In various embodiments,first aperture of camera device 1605 is one of a circular or polygonshaped aperture. The first aperture of camera device 1605 corresponds tocircle 1603. In some other embodiments, the first aperture correspondsto a polygon, e.g., a polygon approximately the same size as circle1603. In some embodiments, the polygon fits inside circle 1603. In someembodiments, the polygon is a regular polygon.

The lenses (1607, 1609, 1611, 1613, 1615, 1617) in said plurality oflenses (1607, 1609, 1611, 1613, 1615, 1617, 1619) which are arrangedalong the perimeter of said first aperture 1603 have optical centers(1621, 1623, 1625, 1627, 1629, 1631) which are arranged along a circle1651. The lenses (1607, 1609, 1611, 1613, 1615, 1617) in said pluralityof lenses (1607, 1609, 1611, 1613, 1615, 1617, 1619) which are arrangedalong the perimeter of said first aperture 1603 have optical centers(1621, 1623, 1625, 1627, 1629, 1631) which form the vertices (corners)of a regular polygon 1655.

The plurality of lenses (1607, 1609, 1611, 1613, 1615, 1617, 1619)includes at least one inner lens 1619 in addition to said lenses (1607,1609, 1611, 1613, 1615, 1617) arranged along the perimeter of said firstaperture 1603. The plurality of lenses (1607, 1609, 1611, 1613, 1615,1617, 1619) includes a total of six lenses (1607, 1609, 1611, 1613,1615, 1617) along the perimeter of said first aperture 1603 and a singlelens (1619) in the center of said six lenses (1607, 1609, 1611, 1613,1615, 1617) arranged along the perimeter of said first aperture 1603.

The non-circular aperture of each of said plurality of lenses (1607,1609, 1611, 1613, 1615, 1617, 1619) is an aperture stop in acorresponding optical chain.

Each lens in said plurality of lenses (1607, 1609, 1611, 1613, 1615,1617, 1619) is part of a corresponding optical chain, each individualoptical chain includes a separate sensor for capturing an imagecorresponding to said individual optical chain.

Apparatus 1605, e.g., a camera device, further includes a module forcombining images captured by separate optical chains into a singlecombined image. In various embodiments, the combining images, e.g.,performed by a module for combining images, includes a shift and addbased on the position of lenses in said plurality of lenses (1607, 1609,1611, 1613, 1615, 1617, 1619).

Camera device 1605 further includes additional elements shown in FIG. 1Aincluding a processor, a memory and a display.

FIG. 8 illustrates a frontal view of the outer lenses of an apparatus1705 implemented in accordance with one embodiment of the presentinvention which incorporates multiple optical chain modules and outerlenses, e.g., the aperture stop lens for each of the correspondingoptical chains, arranged to have non-uniform spacing between the opticalcenters of the lenses. Thus the FIG. 8 embodiment is similar to the FIG.7 embodiment but with non-uniform spacing of the optical centers oflenses along the outer parameter of the lens assembly. The non-uniformspacing facilitates depth of field determinations particularly whenperforming block processing and the entire field of view may not beunder consideration when processing a block or sub-portion of thecaptured field of view. The optical chain modules shown in FIGS. 7 and 8are the same or similar to those previously described with reference toFIG. 3 but differ in terms of lens shape, size and/or configuration. Thedots (1721, 1723, 1725, 1727, 1729, 1731, 1733) in FIG. 8 represent theoptical center of the individual lenses (1707, 1709, 1711, 1713, 1715,1717, 1719), respectively.

FIG. 9 illustrates another exemplary camera device 1801 including aplurality of first through fifth optical chain modules (1890, 1891,1892, 1893, 1894) each of which includes an outer lens (1813, 1815,1817, 1819, 1821), respectively, represented as a circle on the outerlens platter 1803. Each outer lens (1813, 1815, 1817, 1819, 1821) has anoptical axis (1805, 1806, 1807, 1808, 1809), respectively. The opticalaxis (1805, 1806, 1807, 1808, 1809) is represented by an X, indicatingthat the axis goes down into the lens (1813, 1815, 1817, 1819, 1821).The optical axis (1805, 1806, 1807, 1808, 1809), are parallel to eachother. In both FIGS. 18 and 19 arrows made of dashed lines represent thepath of light for the corresponding optical chain module after lightwhich entered the outer lens along the optical axis of the outer lens isredirected by the mirror or other light redirection device. Thus, thearrows represents the direction and general light path towards thesensor of the optical chain to which the arrow corresponds. In variousembodiments, the image deflection element, e.g., a mirror, of theoptical chain changes the direction of the optical rays passing alongthe optical axis of the outer lens by substantially 90 degrees to directthe optical rays passing along the optical axis onto the sensor. Forexample, with regard to optical chain 1890, the image deflection element1823, e.g., a mirror, of the optical chain 1890 changes the direction ofthe optical rays passing along the optical axis 1805 of the outer lens1813 by substantially 90 degrees to direct the optical rays passingalong the optical axis onto the sensor 1853.

In the FIG. 9 embodiment each of the optical chain modules (1890, 1891,1892, 1893, 1894) includes, in addition to an outer lens (1813, 1815,1817, 1819, 1821,) a mirror or other device, e.g., prism, (1823, 1825,1827, 1829, 1831), respectively, for changing the angle of lightreceived via the corresponding outer lens (1813, 1815, 1817, 1819,1821), respectively. Additionally, as in some of the previouslydescribed embodiments such as the FIGS. 1A, 1B, 1C, 1D, and 3embodiments, each optical chain module (1890, 1891, 1892, 1893, 1894),includes a filter (1833, 1835, 1837, 1839, 1841), respectively, and aninner lens (1843, 1845, 1847, 1849, 1851), respectively. In additioneach optical chain module (1890, 1891, 1892, 1893, 1894) includes asensor (1853, 1855, 1857, 1859, 1861), respectively. For example, thefirst optical chain module (OCM 1 1890) include outer lens L1 1813,mirror 1823, filter 1833, inner lens L2 1843 and sensor 1853.

Filters 1833, 1835, 1837, 1839, and 1841 are mounted on a movablecylinder 1875 represented as a circle shown using small dashed lines.The cylinder 1875 may be rotated and/or moved forward or backwardallowing lenses and/or filters on the cylinder to be easily replacedwith other lenses, filter, or holes mounted on the cylinder 1875. Whilein the FIG. 9 example, an exit hole is provided to allow light to exitcylinder 1875 after passing through one of the filters 1833, 1835, 1837,1839, or 1841 it should be appreciated that rather than an exit holeanother lens or filter may be mounted on the cylinder 1875 allowing twoopportunities for the light to be filtered and/or passed through a lensas is passes through the cylinder 1875. Thus, in at least someembodiments a second filter or lens which is not shown in FIG. 18 forsimplicity is included at the exit point for the light as it passesthrough cylinder 1804. Inner lenses are mounted on cylinder 1885 whichis actually closer to the outside sidewalls of the camera device 1801than the filters mounted on cylinder 1875. Given the large diameter ofmovable cylinder 1885 and the relatively small diameter of the lightbeam as it nears the sensor, it should be appreciated that a largenumber of alternative filters, lenses and/or holes can be mounded oncylinder 1885. As with cylinder 1875 the light can be filtered and/orprocessed by a lens as it enters and leaves cylinder 1885 prior toreaching the sensor of the corresponding optical chain.

In some embodiments lenses mounted on a moveable platter positionedbetween the outer lens platter 1803 and mirrors which may, and in someembodiments are, also mounted on a platter are used to supportautofocus. In such an embodiment the lens platter between the outer lensplatter and mirror platter is moved in or out to perform focusoperations for each of the optical chain modules in parallel. In anotherembodiment, different sets of lens are mounted on the drum 1885 or 1875with different lens sets being mounted with a different offset distancefrom the surface of the drum. By switching between the different sets oflenses by rotating the drum on which the different lens sets aremounted, focusing between different predetermined focus set points can,and in some embodiments is achieved, by simply rotating the drum onwhich the lens sets, corresponding to the different focal distance setpoints, are mounted.

Notably, the FIG. 9 embodiment, by changing the direction of lightthrough the use of mirrors, prisms and/or other devices allows for thelength of the individual optical chains to be longer than the cameradevice is thick. That is, the side to side length of the camera device1801 can be used in combination with a portion of the front to backlength to create optical chains having a length longer than the depth ofthe camera device 1801. The longer optical chain length allows for morelenses and/or filters to be used as compared to what may be possiblewith shorter optical chain lengths. Furthermore, the change in thedirection of light allows for the use of cylinders for mounting lenses,filters and/or holes which can be easily interchanged by a simplerotation or axial, e.g., front to back movement, of the cylinder onwhich the lenses, filters and/or holes corresponding to multiple opticalchains are mounted.

In the FIG. 9 embodiment sensors may be fixed and/or mounted on amovable cylinder 1899. Thus, not only can the lenses, filters and/orholes be easily switched, changes between sensors or sets of sensor canbe easily made by rotating the cylinder on which the sensors aremounted. While a single mirror is shown in FIG. 9 in each optical chainmodule, additional mirrors may be used to further extend the length ofthe optical path by reflecting in yet another direction within thehousing of the camera device 1801.

It should be appreciated that the FIG. 9 embodiment allows for acombination of lens, filter, and/or hole mounting platters arrangedparallel with the platter extending left to right within the cameradevice and cylinders arranged so that the top and bottom of the cylinderextend in the front to back direction with respect to the camera body,e.g., with the front of the camera being shown in FIG. 9. Cylinders maybe mounted inside of one another providing a large number ofopportunities to mount lens, filters and/or holes along the opticalpaths of each optical chain module and allowing for a large number ofpossible filter/lens/sensor combinations to be supported, e.g., byallowing for different combinations of cylinder positions for differentmodes of operation.

While changing sensors mounted on a cylinder can be achieved by rotatinga cylinder, in the earlier embodiments in which sensors may be mountedon platters, sensors may be changed by rotating or otherwise moving aplatter on which the sensors are mounted.

Note that in the FIG. 9 embodiment the outer lenses (1813, 1815, 1817,1819, 1821, of the optical chain modules (1890, 1891, 1892, 1893, 1894),respectively, are mounted near the center of the front of the cameradevice 1801 as shown, e.g., forming a generally circular pattern ofouter lenses 1813, 1815, 1817, 1819, 1821.

In camera device 1801 the optical axes (1805, 1806, 1807, 1808, 1809) oflenses (1813, 1815, 1817, 1819, 1821) said optical chain modules (1890,1891, 1892, 1893, 1894) are parallel to each other but at least twomirrors (1823, 1825) corresponding to different optical chains (1890,1891) are not parallel. The light rays of at least two different opticalchains (1890, 1891) cross prior to reaching the sensor (1853, 1855) towhich the rays of said at least two different optical chain modules(1890, 1891) correspond.

In various embodiments, each optical chain module (1890, 1891, 1892,1893, 1894) includes an image deflection element which includes at leastone mirror positioned at 45 degree to said optical axis (1890, 1891,1892, 1893, 1894) of said lens of the optical chain module. For example,with regard to optical chain module 1 1890, in one embodiments, theimage deflection element 1823 is a mirror positioned at 45 degree to theoptical axis 1805 of lens 1813.

In some embodiments, an image deflection element, e.g., image deflectionelement 1823 includes a prism. In some embodiments, an image deflectionelement includes multiple mirrors. In some embodiments, an imagedeflection element includes a combination including at least one mirrorand at least one prism.

FIG. 10 is similar to the FIG. 9 embodiment in that it illustratesanother camera device 1901 including a plurality of optical chainmodules which include mirrors or another device for changing the angleof light entering the optical chain module and thereby allowing at leasta portion of the optical chain module to extend in a direction, e.g., aperpendicular direction, which is not a straight front to back directionwith respect to the camera device. FIG. 10 illustrates another exemplarycamera device 1901 including a plurality of first through fifth opticalchain modules (1990, 1991, 1992, 1993, 1994) each of which includes anouter lens (1913, 1915, 1917, 1919, 1921), respectively, represented asa circle on the outer lens platter 1903. FIG. 10 differs from the FIG. 9embodiment in that the outer lenses (1913, 1915, 1917, 1919, 1921) ofthe first through fifth optical chain modules (1990, 1991, 1992, 1993,1994) are positioned near the perimeter of the face of the camera device1901. This allows for the length of the optical chain module to belonger than the length of the optical chains shown in FIG. 9. FIG. 10shows outer and inner cylinders, also some times referred to as drums,1975, 1985, upon which filters, lenses and holes can and in variousembodiments are mounted as discussed with regard to the FIG. 9embodiment. Thus cylinders 1975 and 1985 server the same or similarpurpose served by cylinders 1875, 1885, respectively. It should beappreciated that in some embodiments the FIG. 10 embodiment includesfilters and lenses mounted on the inner and outer cylinders in the sameor similar manner as filters and lenses are mounted on the cylinders1875, 1885 shown in FIG. 9.

Elements of the FIG. 10 embodiment which are the same or similar to theelements of the FIG. 18 embodiment are identified beginning with “19”instead of “18” and for the sake of brevity will not be described againin detail. For example element 1961 is used to refer to the sensor forthe optical chain module 1994 which includes outer lens 1921,mirror/light redirection device 1931, filter 1941 and inner lens 1951.The cylinder 1975 is used to mount the filters while cylinder 1985 isused to mount the inner lenses.

Each outer lens (1913, 1915, 1917, 1919, 1921) has an optical axis(1905, 1906, 1907, 1908, 1909), respectively. The optical axis (1905,1906, 1907, 1908, 1909) is represented by an X, indicating that the axisgoes down into the lens (1913, 1915, 1917, 1919, 1921). The optical axis(1905, 1906, 1907, 1908, 1909), are parallel to each other.

The camera devices 1801 and 1901 may, and in some embodiments do,include a processor, display and/or other components of the cameradevice shown in FIG. 1A but such elements are not explicitly shown inthe FIGS. 9 and 10 embodiments to avoid complicating the figures andbeing repetitive.

Various functions of the present invention may be and are implemented asmodules in some embodiments. An assembly of modules, e.g., software orhardware modules, may be and are used for performing various functionsof a image processing system or apparatus used to process images inaccordance with embodiments of the present invention. When the modulesare implemented as software modules they may be, and in some embodimentsof the present invention are, stored in memory 108 of FIG. 1A in thesection of memory identified as assembly of modules 118. These modulesmay be implemented instead as hardware modules, e.g., circuits.

The ideas and concepts described with regard to various embodiments suchas those shown in FIG. 10 can be extended so that the input sensors canbe located in a plane, e.g., at the back of the camera device and/or atthe front of the camera device. In some such embodiments the sensors ofmultiple optical chains are mounted on a flat printed circuit board orbackplane device. The printed circuit board, e.g. backplane, can bemounted or coupled to horizontal or vertical actuators which can bemoved in response to detected camera motion, e.g., as part of a shakecompensation process which will be discussed further below. In some suchembodiments, pairs of light diverting devices, e.g., mirrors, are usedto direct the light so that at least a portion of each optical chainextends perpendicular or generally perpendicular to the input and/orsensor plane. Such embodiments allow for relatively long optical pathswhich take advantage of the width of the camera by using mirrors orother light diverting devices to alter the path of light passing throughan optical chain so that at least a portion of the light path extends ina direction perpendicular or generally perpendicular to the front of thecamera device. The use of mirrors or other light diverting devicesallows the sensors to be located on a plane at the rear or front of thecamera device as will now be discussed in detail.

While the invention has been explained using convex lenses in many ofthe diagrams, it should be appreciated that any of a wide variety ofdifferent types of lenses may be used in the optical chain modulesincluding, e.g., convex, concave, and meniscus lenses. In addition,while lenses and filters have been described as separate elements,lenses and filters may be combined and used. For example, a color lensmay, and in some embodiments is, used to both filter light and alter thelights path. Furthermore, while many of the embodiments have beendescribed with a color filter preceding the image sensor of an opticalchain or as using an image sensor with an integrated color filter, e.g.,a Bayer pattern filter, it should be appreciated that use of colorfilters and/or sensors with color filters is not required and in someembodiments one or more optical chain modules are used which do notinclude a color filter and also do not use a sensor with a color filter.Thus, in some embodiments one or more optical chain modules which sensea wide spectrum of color light are used. Such optical chain modules areparticularly well suited for generating black and white images.

In various embodiments image processing is used to simulate a widevariety of user selectable lens bokehs or blurs in the combined imagewith regard to image portions which are out of focus. Thus, whilemultiple lenses are used to capture the light used to generate acombined image, the image quality is not limited to that of anindividual one of the lenses and a variety of bokehs can be achieveddepending on the particular bokeh desired for the combined image beinggenerated. In some embodiments, multiple combined images with differentsimulated bokehs are generated using post image capture processing withthe user being provided the opportunity to save one or more of thegenerated combined images for subsequent viewing and/or printing. Thus,in at least some embodiments a physical result, e.g., a printed versionof one or more combined images is produced. In many if not all casesimages representing real world objects and/or scenes which were capturedby one or more of the optical chain modules of the camera device used totake the picture are preserved in digital form on a computer readablemedium, e.g., RAM or other memory device and/or stored in the form of aprinted image on paper or on another printable medium.

While explained in the context of still image capture, it should beappreciated that the camera device and optical chain modules of thepresent invention can be used to capture video as well. In someembodiments a video sequence is captured and the user can select anobject in the video sequence, e.g., shown in a frame of a sequence, as afocus area, and then the camera device capture one or more images usingthe optical chain modules. The images may, and in some embodiments are,combined to generate one or more images, e.g., frames. A sequence ofcombined images, e.g., frames may and in some embodiments is generated,e.g., with some or all individual frames corresponding to multipleimages captured at the same time but with different frames correspondingto images captured at different times.

While different optical chain modules are controlled to use differentexposure times in some embodiments to capture different amounts of lightwith the captured images being subsequently combined to produce an imagewith a greater dynamic range than might be achieved using a singleexposure time, the same or similar effects can and in some embodimentsis achieved through the use of different filters on different opticalchains which have the same exposure time. For example, by using the sameexposure time but different filters, the sensors of different opticalchain modules will sense different amounts of light due to the differentfilters which allowing different amounts of light to pass. In one suchembodiment the exposure time of the optical chains is kept the same byat least some filters corresponding to different optical chain modulescorresponding to the same color allow different amounts of light topass. In non-color embodiments neutral filters of different darknesslevels are used in front of sensors which are not color filtered. Insome embodiments the switching to a mode in which filters of differentdarkness levels is achieved by a simple rotation or movement of a filterplatter which moves the desired filters into place in one or moreoptical chain modules. The camera devices of the present inventionsupports multiple modes of operation with switching between panoramicmode in which different areas are captured, e.g., using multiple lensesper area, and a normal mode in which multiple lens pointed samedirection are used to capture the same scene. Different exposure modesand filter modes may also be supported and switched between, e.g., basedon user input.

In the FIGS. 11 and 12 embodiments two or more deflection elements areused in each optical chain. Mirrors are exemplary deflection elementsthat may and sometimes are used in the FIGS. 11 and 12 embodiments.Thus, at least in some embodiments each optical chain includes multipledeflection elements in the form of mirrors. In FIGS. 11 and 12embodiments two deflection elements are used in each optical chain witheach deflection element, e.g., mirror, deflecting the light 90 degrees.

FIG. 11 illustrates an exemplary diagram of a camera device 2000implemented in accordance with one exemplary embodiment of theinvention. The FIG. 11 diagram is intended for explanation purposes tofacilitate an understanding of various features and thus is not aprecise view of the camera device as perceived from the top but afunctional diagram of the elements from a top view perspective which isintended to convey various aspects of the optical chain configurationsused in the device 2000. The top portion of FIG. 11 corresponds to thefront of the camera device 2000 while the bottom portion corresponds tothe back of the camera device 2000. The body 2001 of the camera extendsfrom left to right with the lens and/or openings 2002, 2004, 2006corresponding to multiple optical chains being mounted in front portion2010 of the camera device 2000. A LCD or other display (not shown) mayand in some embodiments is, located at the rear of the camera device2000.

In the camera device 2000 includes a plurality of lens or openings L1through LZ 2002, 2004, 2006 each corresponding to a different one of Zoptical chains. Note that in FIG. 11 the lenses 2002, 2004 and 2006 areloaded in a plane represented by dashed line 2012 which extends downtowards the bottom of the camera device 2000 which is not visible in theFIG. 11 diagram. The lenses 2002, 2004, and 2006 may be arranged in acircular or other pattern on the front of the camera device 2002. Eachoptical chain in the FIG. 11 embodiment includes multiple mirrors orother light redirecting devices and a sensor positioned at the end ofthe optical chain. For example, optical chain 1 includes lens 2002,first mirror 2022, second mirror 2024 and sensor 2038. Optical chain Zincludes lens LZ 2006, first mirror 2028, second mirror 2026 and sensorZ 2034. It should be appreciated that mirrors of the first and secondoptical chains are located around the cylinder 2020 on which one or morelenses or filters may be mounted as discussed with regard to the otherembodiments. The mirrors may be arranged in a plane positioned parallelto the input plane 2012 with the light of the different optical chainspassing each other, e.g., crossing, within the cylinder 2020. While asingle cylinder 2020 is shown in FIG. 11, multiple cylinders, lensesand/or filters may, and in some embodiments are, used as discussed withregard to the other embodiments. Note that in the FIG. 11 embodiment themirrors (2022, 2024), (2028, 2026) redirect the light passing throughthe optical chain to which the mirrors correspond so that at least aportion of the optical path of the optical chain extends perpendicularor generally perpendicular to the input direction in which the inputlenses L1, L2, LZ face and parallel to the input plane 2012. The inputplane may be implemented as a mounting device, e.g., circuit board, uponone or more input lenses or openings L1, L2, LZ are mounted or includedin. This allows the optical chain to take advantage of the left to rightwidth of a camera permitting an overall optical chain length than wouldbe possible if the optical chain was limited to the front to back depthof camera device 2000. This allows for thin cameras with relatively longoptical chains. Notably, the use of two 45 degree mirrors 2022, 2024allows the sensors of the optical chain to be mounted in a backplane2030 with the sensors being arranged on the backplane 2030 in a planewhich is parallel to the input plane 2012. The ability to mount thesensors on a single backplane allows for the simple movement of thesensors as an assembly maintaining the relative position of the sensors2034, 2038 to one another on the backplane 2030 even if the backplane ismoved. The cylinder and mirrors may, but need not be, mounted in amanner so that they will move with the backplane 2030 maintaining thealignment of the optical chains to one another as the backplane 2030 ismoved, e.g., up or down or left to right in the camera body 2000. Thus,in some embodiments the backplane 2030 and sensors 2034, 2038 can bemoved in unison, e.g., by applying a force to the backplane 2030 toinduce motion as may be desired.

In one embodiment, motion sensors 2040 are included in the camera device2000. The motion sensors 2040 may be accelerometers and/or gyroscopesused to detect motion along one or more axis of the camera. In oneparticular embodiment a shake compensation module 2042 is included inthe camera device 2000. The shake compensation module 2042 receivesoutput from the motion sensors 2040 and detects camera movement, e.g.,movement indicative of un-intentional shaking as is common in the caseof hand held cameras. The shake compensation control module is coupledto a horizontal actuator 2032 and a vertical actuator 2036 which are incontact with the backplane 2030 which may be a circuit board. Thevertical actuator 2036 is shown in dashed lines since it is positionedbelow backplane 2030 and would not be visible from the top. The verticalactuator 2036 can be used to move the backplane 2030, e.g. circuitboard, up or down while actuator 2032 can be used to move the backplane2030 left or right. In at least one embodiment backplane 2030 is mountedin a manner that allows motion left and right, up and down, but whichmaintains its parallel relationship to the input plane 2012. In someembodiments backplane 2030 is mounted in a slot which is part of thehousing of the camera device 2000. The actuators 2032, 3036 may bemotorized or implemented using elements which expand or contract when avoltage is supplied. The shake compensation control module 2042 controlsthe supply of power and/or control signals to actuators 2032, 2036 whichinduces motion of the backplane 2030 and sensors mounted thereon whichis intended to counteract the shaking. The motion of the backplane 2030is normally not detectable to the holder of the camera but can reducethe distorting in the captured images induced by shaking of the camerahousing in which the various elements of the camera are mounted. Thelenses and/or openings 2002, 2004, 2006 may not distort or focus theincoming light and may remain fixed while one or more of the otherelements of the optical chains move, e.g., to compensate for shakingand/or changes the lenses on the cylinder or drum 2020 through whichlight will pass.

The FIG. 11 embodiment is particular well suited for embodiments whereit is desirable from a manufacturing standpoint and/or shakecompensation standpoint to mount the sensors 2034, 2038 on backplanessuch as printed circuit boards or other relatively flat mounting deviceswhether they be out of metal, plastic, another material or a combinationof materials.

It should be appreciated that the camera device 2000, as well as thecamera device 2100 shown in FIG. 12 may include the elements of thecamera device 100 shown in FIG. 1A in addition to those shown in FIGS.11 and 12 but that such elements are omitted to facilitate anunderstanding of the elements and configuration which is explained usingFIGS. 11 and 12.

FIG. 12 illustrates an additional exemplary camera device 2100 in whichmirrors (2122, 2124), (2128, 2126) and/or other light redirectingelements are used to alter the path of light in the optical chains sothat the input light input lenses and/or opens can be arranged in one ormore planes at the front of the camera where the lens and/or openingsthrough which light enters the optical chains are also located. Elementsin FIG. 12 which are the same or similar the elements of FIG. 11 arenumbered using similar numbers but starting with the first two digits 21instead of 20. Such similar elements will not be described again expectto point out some of the differences between the FIG. 12 and FIG. 11configurations.

One of the important differences between the devices 2100 and 2000 isthat in the camera device 2100 both the sensors 2134, 2138 and externallenses/openings of the optical chains are located in the front of thecamera. This is made possible by having the second mirror 2124 or 2126direct light to the front of the camera rather than the back of thecamera. In the FIG. 12 embodiment the input plane and the sensor planemay be the same plane or positioned in close proximity to each other. Asin the case of the FIG. 11 embodiment vertical and horizontal actuators2132, 2136 may be provided and used to mechanically compensate fordetected camera shaking.

The FIG. 12 embodiment may be desirable where a manufacturer may want tocombine the input plane assembly and sensor plane assembly into a singleunit as part of the manufacturing processor prior to combining it withthe cylinder/lens assembly 2120.

Numerous variations on the designs shown in FIGS. 11 and 12 arepossible. Significantly, the methods and apparatus of the presentinvention allow for sensors to be arranged parallel to or on anyinternal wall of a camera device while still allowing for a cameradevice to include multiple optical chains in a relatively thin camera.By configuring the sensors parallel to the front or rear walls of thecamera rather than the side walls, the sensors and/or lens can be spreadout and occupy a greater surface area than might be possible if thecamera sensors were restricted to the sidewalls or some otherarrangement.

Notably many of the embodiments are well suited for allowing a LCD orother display to be placed at the back of the camera facing out withoutthe display panel significantly interfering with the overall length ofthe individual optical chain modules included in the camera.

While the invention has been explained using convex lenses in many ofthe diagrams, it should be appreciated that any of a wide variety ofdifferent types of lenses may be used in the optical chain modulesincluding, e.g., convex, concave, and meniscus lenses. In addition,while lenses and filters have been described as separate elements,lenses and filters may be combined and used. For example, a color lensmay, and in some embodiments is, used to both filter light and alter thelights path. Furthermore, while many of the embodiments have beendescribed with a color filter preceding the image sensor of an opticalchain or as using an image sensor with an integrated color filter, e.g.,a Bayer pattern filter, it should be appreciated that use of colorfilters and/or sensors with color filters is not required and in someembodiments one or more optical chain modules are used which do notinclude a color filter and also do not use a sensor with a color filter.Thus, in some embodiments one or more optical chain modules which sensea wide spectrum of color light are used. Such optical chain modules areparticularly well suited for generating black and white images.

In various embodiments image processing is used to simulate a widevariety of user selectable lens bokehs or blurs in the combined imagewith regard to image portions which are out of focus. Thus, whilemultiple lenses are used to capture the light used to generate acombined image, the image quality is not limited to that of anindividual one of the lenses and a variety of bokehs can be achieveddepending on the particular bokeh desired for the combined image beinggenerated. In some embodiments, multiple combined images with differentsimulated bokehs are generated using post image capture processing withthe user being provided the opportunity to save one or more of thegenerated combined images for subsequent viewing and/or printing. Thus,in at least some embodiments a physical result, e.g., a printed versionof one or more combined images is produced. In many if not all casesimages representing real world objects and/or scenes which were capturedby one or more of the optical chain modules of the camera device used totake the picture are preserved in digital form on a computer readablemedium, e.g., RAM or other memory device and/or stored in the form of aprinted image on paper or on another printable medium.

While explained in the context of still image capture, it should beappreciated that the camera device and optical chain modules of thepresent invention can be used to capture video as well. In someembodiments a video sequence is captured and the user can select anobject in the video sequence, e.g., shown in a frame of a sequence, as afocus area, and then the camera device capture one or more images usingthe optical chain modules. The images may, and in some embodiments are,combined to generate one or more images, e.g., frames. A sequence ofcombined images, e.g., frames may and in some embodiments is generated,e.g., with some or all individual frames corresponding to multipleimages captured at the same time but with different frames correspondingto images captured at different times.

While different optical chain modules are controlled to use differentexposure times in some embodiments to capture different amounts of lightwith the captured images being subsequently combined to produce an imagewith a greater dynamic range than might be achieved using a singleexposure time, the same or similar effects can and in some embodimentsis achieved through the use of different filters on different opticalchains which have the same exposure time. For example, by using the sameexposure time but different filters, the sensors of different opticalchain modules will sense different amounts of light due to the differentfilters which allowing different amounts of light to pass. In one suchembodiment the exposure time of the optical chains is kept the same byat least some filters corresponding to different optical chain modulescorresponding to the same color allow different amounts of light topass. In non-color embodiments neutral filters of different darknesslevels are used in front of sensors which are not color filtered. Insome embodiments the switching to a mode in which filters of differentdarkness levels is achieved by a simple rotation or movement of a filterplatter which moves the desired filters into place in one or moreoptical chain modules. The camera devices of the present inventionsupports multiple modes of operation with switching between panoramicmode in which different areas are captured, e.g., using multiple lensesper area, and a normal mode in which multiple lens pointed samedirection are used to capture the same scene. Different exposure modesand filter modes may also be supported and switched between, e.g., basedon user input.

FIG. 13 is a flowchart 2600 of an exemplary method of generating videofrom a sequence of image data captured by a camera moving along a pathin accordance with an exemplary embodiment. Operation starts in step2602 and proceeds to step 2603. In step 2603 the camera captures imagedata and related movement data over a period of time. Operation proceedsfrom step 2603 to step 2604.

In step 2604 camera motion is detected, e.g., the path of motion of themoving camera is tracked, said moving camera includes multiple opticalchains or being a light field camera, e.g., a Lytro camera, said movingcamera supporting image synthesis from any of a plurality of points ofview within a synthetic aperture region, e.g., a set of all the pointsof view from which an image can by synthesized by the camera, of saidcamera. In some embodiments, step 2604 includes step 2606 in which atleast one of an accelerometer and gyroscope included in said camera ismonitored. The output of such devices is used to detect motion. Inanother embodiment step 2604 includes comparing images captured by thecamera device to detect motion. For example an image captured at a firsttime may be compared to an image captured at a second time to detectmotion. The compared images maybe from a camera module, e.g., opticalchain, of the camera which captures the images at two different times, afirst frame time and a second frame time. Motion maybe and sometimes isdetermined from a change in position of an object in the images whichare compared where the object maybe a stationary object such as a treewhich is part of a background portion of the image. Motion informationmay be, and sometimes is stored with captured images in memory so thatit is available for use in generating one or more composite images at alater time. In the case where camera motion information is generatedfrom the captured images, the motion information may be generatedoutside the camera device, e.g., by a device which processes capturedimages, e.g., prior to generating a composite image from the capturedimages.

Operation proceeds from step 2604 to step 2608. In step 2608, a trackstabilization operation is performed. Step 2608 includes steps 2610 andstep 2620. In some embodiments, step 2608 further includes step 2616. Insome such embodiments, step 2608 further includes step 2618.

In step 2610 a sequence of points of view to be used for synthesizing asequence of images of said video based on said path of motion isdetermined. In some embodiments, step 2610 includes steps 2612 and 2614.In step 2612 a smoothing operation on the tracked path of motion isperformed to produce a smoothed path of motion, and in step 2614 thesmoothed path is used to determine said sequence of points of view. Insome embodiments, the smoothed path is a straight line. In some otherembodiments, the smoothed path is a smooth curve. In some embodiments,operation proceeds from step 2610 to step 2620. In other embodiments,e.g., an embodiment including step 2616, operation proceeds from step2610 to step 2618.

In step 2616 images captured by the camera are cropped based on aninadvertent motion, e.g., rotation and/or linear motion, of the camera.Operation proceeds from step 2616 to step 2618. In step 2618 adjustmentsare performed to the cropping of said images and adjustments areperformed to the sequence of points of view based on measurementsgenerated by a least one of a gyroscope or an accelerometer included insaid camera. Operation proceeds from step 2618 to step 2620.

In step 2620 said sequence of images is synthesized, said synthesizedsequence of images based the determined sequence of points of view. Insome embodiments, each image in the sequence being synthesized fromcaptured image outputs by multiple different optical chain modules,e.g., at the same time, with the synthesized center of the generatedimage being a function of a determined point of view to be used ingenerating the synthesized image. In some embodiments, the synthesizedsequence of images has the determined sequence of points of view. Insome other embodiments, the synthesized sequence of images has theadjusted determined sequence of points of view.

Operation proceeds from step 2608 to step 2622. In step 2622 thesynthesized sequence of images is output as said video. Operationproceeds from step 2622 to step 2603.

In some embodiments, the synthetic aperture of the camera issufficiently large to include expected points of view corresponding to arange of inadvertent track deviation expected to be encountered by ahandheld camera. In some embodiments, the camera is a handheld camerathat is manually moved along said path, and the path is a straight line.In various embodiments, the camera is a handheld camera and can bemanually moved along a straight path providing the same or similarresults to a camera mount and moved on a track.

In some embodiments, the camera is a portable camera that is mounted ona vehicle and the vehicle is driven or moved along said path, e.g., astraight path or a curved path. In some such embodiments, the vehicleand camera mount does not include inertial stabilization, e.g., thecamera is not mounted on an initially stabilized platform. In variousembodiments, the camera provides the same or similar results to a cameramounted on an inertially stabilized platform on a moving vehicle. Insome embodiments, the vehicle on which the camera is mounted is anunmanned vehicle.

In some embodiments, each of the steps of flowchart 2600 are implementedby a camera device including multiple optical chains or being a lightfield camera, e.g., a Lytro camera. In some embodiments, the cameradevice, which includes multiple optical chains, or which is a lightfield camera, e.g. a Lytro camera, is a cell phone or other portablecamera device, e.g., an electronic tablet, electronic pad, webcamdevice, surveillance device, etc. In one exemplary embodiment, cameradevice 2800 of FIG. 15 implements the steps of the method of flowchart2600.

In other embodiments, some steps of flowchart 2600 are implemented by adevice, e.g., a computer system, e.g., computer system 1400 of FIG. 6,external to the camera device, and the other steps of flowchart 2600 areimplemented by the exemplary camera device including multiple opticalchains or being a light field camera, e.g., a Lytro camera, e.g., cameradevice 2800 of FIG. 15.

FIG. 14 is a drawing of an assembly of modules 2700, which may beincluded in camera device 2800 of FIG. 15 and/or in a camera deviceimplementing a method in accordance with flowchart 1300 of FIG. 14.Assembly of modules 2700 includes a module 2703 configured to captureimage data and related movement data over a period of time, a module2704 configured to track the path of motion of a moving camera, saidmoving camera including multiple optical chains or being a light fieldcamera, said moving camera supporting image synthesis from any of aplurality of points of view within a synthetic aperture region of saidcamera. Module 2704 includes a module 2706 configured to monitor anoutput of at least one of an accelerometer and gyroscope included insaid camera. Assembly of modules 2700 further includes a module 2708configured to perform a track stabilization operation. Module 2708includes a module 2710 configured to determine a sequence of points ofview to be used for synthesizing a sequence of images of said videobased on said path of motion and a module 2716 configured to crop imagescaptured by the camera based on inadvertent motion of the camera, e.g.,inadvertent angular and/or inadvertent linear motion.

In some embodiments, module 2710 determines one point of view in thesequence being generated for each synthesized image to be produced andincluded in the sequence of images. Module 2710 includes a module 2712configured to determine a sequence of points of view to be used forsynthesizing a sequence of images of said video based on said path ofmotion and a module 2714 configured to use said smoothed path todetermine said sequence of points of view. Assembly of modules 2708further includes a module 2718 configured to perform adjustments to thecropping of said images and adjustment to the sequence of points of viewbased on measurements generated by at least one of a gyroscope and anaccelerometer included in said camera, and a module 2720 configured tosynthesize said sequence of images, said synthesized sequence of imagesbased on said determined sequence of points of view. Assembly of modules2700 further includes a module 2722 configured to output saidsynthesized sequence of images as said video.

In some embodiments, a module shown in assembly of modules 2700 as beingincluded within another module may be implemented as a separate module,e.g., an independent module.

FIG. 15 illustrates an exemplary apparatus 2800, sometimes referred tohereinafter as a camera device, implemented in accordance with oneexemplary embodiment of the present invention. The camera device 2800,in some embodiments, is a portable device, e.g., a cell phone or tabletincluding a camera assembly. In various embodiments camera device 2800is a handheld device. Camera device 2800 is an image capture devicewhich includes a plurality of camera modules, e.g., a plurality ofoptical chain modules 2830 and/or a light field camera module 2862,e.g., a LYTRO module. In various embodiments, camera device 2800 isconfigured to capture a sequence of image data as the camera device 2800is moved along a path.

In various embodiments, the camera device 2800 is configured to supportoperation as a moving camera that supports image synthesis from any of aplurality of points of view within a synthetic aperture region. In somesuch embodiments, the synthetic aperature region is a region whichcorresponds to multiple camera modules included in said camera. In somesuch embodiments, the synthetic aperature region is a circular regionhaving a diameter approximately the diameter of a circular areaincluding the outermost lens of the multiple camera modules.

FIG. 15 illustrates the camera device 2800 in block diagram form showingthe connections between various elements of the apparatus 2800. Theexemplary camera device 2800 includes a display device 2802, an inputdevice 2806, memory 2808, a processor 2810, a transceiver interface2814, e.g., a cellular interface, a WIFI interface, or a USB interface,an I/O interface 2812, and a bus 2816 which are mounted in a housingrepresented by the rectangular box touched by the line leading toreference number 2800. The input device 2806 may be, and in someembodiments is, e.g., a keypad, a touch screen, or similar device thatmay be used for inputting information, data and for instructions. Thedisplay device 2802 may be, and in some embodiments is, a touch screen,used to display images, video, information regarding the configurationof the camera device, and/or status of data processing being performedon the camera device. In the case where the display device 2802 is atouch screen, the display device 2802 serves as an additional inputdevice and/or as an alternative to the separate input device, e.g.,buttons, 2806. The I/O interface 2812 couples the display 2802 and inputdevice 2806 to the bus 2816 and interfaces between the display 2802,input device 2806 and the other elements of the camera which cancommunicate and interact via the bus 2816. In addition to being coupledto the I/O interface 2812, the bus 2816 is coupled to the memory 2808,processor 2810, an optional autofocus controller 2832, a transceiverinterface 2814, and, in some embodiments, a plurality of optical chainmodules 2830, e.g., N optical chain modules. The optical chain modulesmay, and in some embodiments are, the same as or similar to any of thoseshown in the FIGS. 1B, 1C, 1D, 2, 3, 4, 7 and 8 embodiments. In someembodiments N is an integer greater than 2, e.g., 3, 4, 7 or a largervalue depending on the particular embodiment. Images captured byindividual optical chain modules in the plurality of optical chainmodules 2830 can be stored in memory 2808, e.g., as part of thedata/information 2820 and processed by the processor 2810, e.g., togenerate one or more composite images. Multiple captured images and/orcomposite images may be processed to form video, e.g., a series ofimages corresponding to a period of time. In some embodiments, cameradevice 2800 includes light field module 2862, e.g., a Lytro module forcapturing a light field, coupled to bus 2816.

As should be appreciated the effective aperture of the camera device maybe smaller or as large as the combination of apertures which are part ofthe camera device 2800. Thus, for example, when the optical cameramodules having the arrangement shown in FIG. 7 or 8 are used theeffective aperture maybe the same or smaller than the area occupied bythe seven lenses which are included in the camera device 2800. Asimulated aperture may be smaller than the combined aperture of theindividual camera modules. This may be the result of cropping of one ormore captured images which are combined. The point of view of thesimulated aperture can be controlled as part of the process of combiningthe images of the multiple different capture modules. Thus, the point ofview of the combined image need not correspond to the center of the setof optical modules which are included in the camera device 2800. As thecamera device moves unintentional up or down for instance, the point ofview may change. However, in the image combining process the point ofview for different sequential frames may, and in some embodiments is,controlled so that the point of view in the sequential combined imagesappears as though it follows a smooth track of motion such as that whichmight be expected if the camera device capturing the image sequence wason an actual physical track rather than being handheld and subject tounintended motion or deviation from a smooth path.

Camera device 2800 further includes an inertial measurement module 2852,for measuring camera motion, coupled to bus 2816. Inertial measurementmodule 2852 includes a plurality of gyroscopes (gyroscope 1 2854, . . ., gyroscope N 2856), and a plurality of accelerometers (accelerometer 12858, . . . , accelerometer N 2860). In some embodiments, there aresufficient gyroscopes to measure angular rate on three orthogonal axis,e.g., three single axis gyroscopes, two dual axis gyroscopes, or onedual axis gyroscopes and one single axis gyroscope. In variousembodiments, there are sufficient accelerometers to measure accelerationalong 3 axis, e.g., three accelerometers mounted in a triad, with threesubstantially orthogonal accelerometer measurement axis. In someembodiments, the inertial measurement module 2852 is included in asingle chip or portion of a single chip.

Transceiver interface 2814 couples the internal components of the cameradevice 2800 to an external network, e.g., the Internet, and/or one ormore other devices e.g., memory or stand alone computer. Via interface2814 the camera device 2800 can and does output data, e.g., capturedimages, generated composite images, and/or generated video. The outputmay be to a network or to another external device for processing,storage and/or to be shared. The captured image data, generatedcomposite images and/or video can be provided as input data to anotherdevice for further processing and/or sent for storage, e.g., in externalmemory, an external device or in a network.

The transceiver interface 2814 of the camera device 2800 may be, and insome instances is, coupled to a computer so that image data may beprocessed on the external computer. In some embodiments the externalcomputer has a higher computational processing capability than thecamera device 2800 which allows for more computationally complex imageprocessing of the image data outputted to occur on the externalcomputer. The transceiver interface 2814 also allows data, informationand instructions to be supplied to the camera device 2800 from one ormore networks and/or other external devices such as a computer or memoryfor storage and/or processing on the camera device 2800. For example,background images may be supplied to the camera device to be combined bythe camera processor 2810 with one or more images captured by the cameradevice 2800. Instructions and/or data updates can be loaded onto thecamera via interface 2814 and stored in memory 2808.

The camera device 2800 may include, and in some embodiments doesinclude, an autofocus controller 2832 and/or autofocus drive assembly2834. The autofocus controller 2832 is present in at least someautofocus embodiments but would be omitted in fixed focus embodiments.The autofocus controller 2832 controls adjustment of at least one lensposition in the optical chain modules used to achieve a desired, e.g.,user indicated, focus. In the case where individual drive assemblies areincluded in each optical chain module, the autofocus controller 2832 maydrive the autofocus drive of various optical chain modules to focus onthe same target. As will be discussed further below, in some embodimentslenses for multiple optical chain modules are mounted on a singleplatter which may be moved allowing all the lenses on the platter to bemoved by adjusting the position of the lens platter. In some suchembodiments the autofocus drive assembly 2834 is included as an elementthat is external to the individual optical chain modules with the driveassembly 2834 driving the platter including the lenses for multipleoptical chains under control of the autofocus controller 2832. While theoptical chain modules will in many embodiments be focused together tofocus on an object at a particular distance from the camera device 2800,it is possible for different optical chain modules to be focused todifferent distances and in some embodiments different focus points areintentionally used for different optical chains to increase the postprocessing options which are available.

The processor 2810 controls operation of the camera device 2800 tocontrol the elements of the camera device 2800 to implement the steps ofthe methods described herein. The processor may be a dedicated processorthat is preconfigured to implement the methods. However, in manyembodiments the processor 2810 operates under direction of softwaremodules and/or routines stored in the memory 2808 which includeinstructions that, when executed, cause the processor to control thecamera device 2800 to implement one, more or all of the methodsdescribed herein. Memory 2808 includes an assembly of modules 2818wherein one or more modules include one or more software routines, e.g.,machine executable instructions, for implementing the image captureand/or image data processing methods of the present invention.Individual steps and/or lines of code in the modules of 2818 whenexecuted by the processor 2810 control the processor 2810 to performsteps of the method of the invention. When executed by processor 2810,the data processing modules 2818 cause at least some data to beprocessed by the processor 2810 in accordance with the method of thepresent invention. The resulting data and information are stored in datamemory 2820 for future use, additional processing, and/or output, e.g.,to display device 2802 for display or to another device fortransmission, processing and/or display. The memory 2808 includesdifferent types of memory for example, Random Access Memory (RAM) inwhich the assembly of modules 2818 and data/information 2820 may be, andin some embodiments are stored for future use. Read only Memory (ROM) inwhich the assembly of modules 2818 may be stored for power failures.Non-volatile memory such as flash memory for storage of data,information and instructions may also be used to implement memory 2808.Memory cards may be added to the device to provide additional memory forstoring data (e.g., images and video) and/or instructions such asprogramming. Accordingly, memory 2808 may be implemented using any of awide variety of non-transitory computer or machine readable mediumswhich serve as storage devices.

In one embodiment the assembly of modules 2700 shown in FIG. 14 is partof or used in place of the assembly of modules 2818. The modules in theassembly 2700, when executed by the processor 2810 control the cameradevice 2800 in one embodiment to implement the method described withregard to FIG. 13. While the modules of assembly of modules 2700 of FIG.14 may, and in some embodiments are implemented using software, in otherembodiments they are implemented in hardware, e.g., as circuits, whichmay and in some embodiments are included in the camera device 2800,e.g., as assembly of modules 2880.

In another embodiment, some of the modules of assembly of modules 2700are included as part of assembly of modules 2818 and/or assembly ofmodules 2880 of camera device 2800 and some of the modules of assemblyof modules 2700 are included as part of assembly of modules 1418 ofcomputer system 1400. For example, in one exemplary embodiment, imagedata collection and camera motion measurements are be performed by thecamera device 2800 and processing of the collected data and collectedcamera motion measurements is performed by computer system 1400. Inother embodiments, some steps of the processing are performed by thecamera device 2800 and other steps performed by the computer system1400.

In some embodiments, camera device 2800 includes a path tracking module2870 which detects camera motion, e.g., implemented as circuitry, and atrack stabilization apparatus 2870. In some such embodiments, the trackstabilization apparatus includes a point of view determination module2874, a synthesization module 2876 and an output module 2878. Pathtracking module 2870 is configured to track the path of the cameradevice 2800 which is an image capture device. Point of viewdetermination module 2874 is configured to determine a sequence ofpoints of view to be used for synthesizing a sequence of images of videobased on the path of motion. Synthesization module 2876 is configured tosynthesize a sequence of images, said synthesized sequence of imagesbeing based on the determined points of view. Output module 2878 isconfigured to output a synthesized sequence of images as video.

Drawing 2900 of FIG. 16 illustrates a camera mounted on a moveablecamera cart 2908 with a single optical chain moving along a straightline track with rails 2906 in the direction of motion 2903. The path ofmotion of the camera is indicated by dotted line 2902. The camera attime T0 is shown as element 2904; the camera at time T1 is shown aselement 2904′ and the camera at time T2 is shown as element 2904″.

Drawing 2950 of FIG. 16 illustrates an exemplary handheld camera, inaccordance with an exemplary embodiment of the present invention, withmultiple optical chains, with gyroscopes and with accelerometers, beingmoved. The information generated by such measurement devices can, and insome embodiments is, used to determine a path of motion and what pointof view should be used, in some embodiments, for purposes of combiningimage data from different optical chain modules of the camera.

Exemplary person 2958 is holding the camera, including multiple opticalchains, gyroscopes and accelerometers, and moving, e.g., walking, alonga straight line path with path borders 2956 in the direction of motion2903. In one example, the path borders 2956 are at the same location asrails 2906. The camera, including multiple optical chains, gyroscopesand accelerometers, at time T0 is shown as element 2954; the camera attime T1 is shown as element 2954′ and the camera at time T2 is shown aselement 2954″. The nominal path of motion of the camera is indicated bydotted line 2902. There are inadvertent motions of the camera, e.g.,rotations and linear motions, as the handheld camera is moved along thepath 2902. The path of motion of the moving camera is tracked, atracking stabilization operation is performed, and a synthesizedsequence of images is output as video, e.g., in accordance with themethod of flowchart 2600 of FIG. 26. The synthesized sequence of imagemay include a serious of images generated from images captured atdifferent times.

In one embodiment while each image in a sequence corresponds to adifferent, e.g., sequential time period, the generated imagecorresponding to one time period in the sequence is generated fromcaptured image data which is captured by multiple different individualoptical chain modules operating in parallel during the time period towhich the individual generated image corresponds. As part of the imagegeneration process, the images captured by different optical chainmodules corresponding to an individual time period may be combined basedon a point of view that is determined based on camera motion. The pointof view used from one frame to the next is selected in some embodimentsto provide the appearance of a consistent or smoothly changing point ofview as opposed to relying on a center portion of the camera device orset of optical chain modules as the point of view. Thus, the point ofview used for controlling the combining process maybe different from thepoint of view of the individual optical chain modules used to capturethe image data being combined. In this manner, as part of the combiningoperation the point of view may be adjusted as may be necessary tosimulate a smooth track of motion. Image cropping may be used as part ofthe combining operation as well to ensure that area included in theoutput video sequence remains relatively consistent and changesgradually over time as might be expected by smooth intentional cameramotion as opposed to inadvertent motion, which often takes the form ofjerky camera motion, that may be the result of the use of a handheldcamera device. Thus, by using a large synthetic aperture, e.g.,simulated aperture generated by using multiple smaller apertures incombination and by outputting an image smaller than the maximum imagesize which may be captured, image adjustments in the form of croppingand altering the point of view used for generating an image can be usedto reduce or eliminate the effect of unintended motion as a cameradevice is moved along a path, e.g., a path which is intended to besmooth but may be jerky or subject to unintentional changes in theactual point of view of individual optical chain modules as a result ofunintended motion.

The exemplary camera device of drawing 2950 is in some embodiments,camera device 2800 of FIG. 15+.

FIG. 17, comprising the combination of FIG. 17A, FIG. 17B, FIG. 17C andFIG. 17D, is a flowchart 3000 of an exemplary method of generatingimages in accordance with an exemplary embodiment. In some embodiments,the exemplary method of flowchart 3000 is performed by a camera deviceincluding multiple optical chains, e.g., camera device 2800 of FIG. 15.In other embodiments, some steps of flowchart 3000 are implemented by acamera device including multiple optical chains, e.g., camera device2800 of FIG. 15, and other steps of flowchart 3000 are implemented by adevice external to the camera device, e.g., a computer system such ascomputer system 1400 of FIG. 6. In one such embodiment, the capturedimages from the multiple optical chains of the camera device areprocessed externally from the camera device by a computer system whichgenerate the composite images.

The exemplary method of flowchart 3000 will be described for anembodiment in which an exemplary camera device performs each of thesteps; however, it should be appreciated that some of the steps offlowchart 3000, e.g., image processing steps, may be, and in someembodiments, are performed by another device, e.g., a computer system.

Operation of the exemplary method begins in step 3002 in which thecamera device is powered on and initialized. Operation proceeds fromstep 3002 to step 3004, in which the camera device initializes thecurrent image capture time period T to 1. Operation proceeds from step3004 to step 3006, and in some embodiments to step 3003.

In step 3003, the camera device monitors for camera motion, e.g., usinggyroscopes and accelerometers. Operation proceeds from step 3003 to step3005, in which the camera device stores motion information correspondingto captured images, e.g., information indicating motion between the lastcapture time period (T=1) and the current image capture time period (T),with images captures in the current image capture time period. Operationproceeds from step 3005 to step 3003. The monitoring for motion andstoring of motion information is performed on an ongoing basis.

Returning to step 3006, in step 3006 the camera device captures imagesduring a first image capture time period (T=1) using multiple opticalchains of the camera device. Operation proceeds from step 3006 to step3008. In step 3008, the camera device stores images captured during thefirst image capture time period. Operation proceeds from step 3008 tostep 3010. In step 3010 the camera device generates a composite imagefrom images captured during the first time period using a firstreference image. In some embodiments, the first reference image is animage captured by one of the optical chains, e.g., a center opticalchain. In some embodiments, the first composite image is an image havingthe same perspective as the first reference image. Operation proceedsfrom step 3010 to step 3011.

In step 3011 the camera device stores, displays, and/or outputs thegenerated composite image. Operation proceeds from step 3011 to step3012. In step 3012 the camera device updates the current image capturetime period by one, e.g., sets T=T+1. Operation proceeds from step 3012to step 3104. In step 3104, the camera device captures imagescorresponding to the current image capture time period (T), e.g., thesecond time period. Operation proceeds from step 3014 to step 3016. Instep 3106, the camera device stores images corresponding to the currentimage capture time period, e.g., the second time period. Operationproceeds from step 3016, via connecting node A 3018, to step 3020.

In step 3020 the camera device generates a composite image for thecurrent image capture time period, e.g., the second time period. Step3020 includes step 3022, step 3032, and step 3044. In step 3022 thecamera device detects an amount of motion. Step 3024 includes one orboth of steps 3024 and 3026. In step 3024 the camera device detects anamount of motion of the camera device including multiple optical chains,e.g., motion between the first time period and the second time period.In various embodiments, the detection of step 3024 is based upon thestored motion information from step 3005, e.g., based on gyroscopesand/or accelerometer measurements.

In step 3026, the camera device detects an amount of motion between animage corresponding to the second time period and an image correspondingto the first time period. In some embodiments, step 3026 includes step3028 in which the camera device compares the content of an imagecorresponding to the first time period to an image corresponding to thesecond time period. In some embodiments, step 3028 includes step 3030 inwhich the camera device compares a first image captured by a firstoptical chain module of said camera during the first time period to asecond image captured by the first optical chain module during saidsecond time period.

Operation proceeds from step 3030 to step 3032. In step 3032 the cameradevice produces a second reference image from a first plurality ofimages captured by different optical chains of said camera device duringthe second time period. In some embodiments, the first plurality ofimages is, e.g., a plurality of frames with one frame per optical chain.Step 3032 includes step 3034 or step 3036.

In step 3034 the camera device selects a reference image from imagescaptured by different optical chain modules during the second timeperiod based on the detected amount of motion. In some embodiments, step3034 includes step 3038 or step 3040. In step 3038 the camera deviceselects the second reference image based on detected motion of thecamera including a plurality of optical chain modules having differentperspectives, different ones of said plurality of images being capturedby different ones of said plurality of optical chain modules. In someembodiments step 3038 includes step 3039. In step 3039, the cameradevice selects as the second reference image an image captured by anoptical chain of said camera having a second perspective which is theclosest to a first perspective of an optical chain of said camera thatwas used to capture the first reference image.

In step 3040 the camera device selects the second reference image basedon detected motion in images captured by the optical chain modules ofthe camera device, said motion being detected by comparing the contentof an image corresponding to said first time period to an imagecorresponding to said second time period. In some embodiments, step 3040includes step 3041. In step 3041, the camera device selects as thesecond reference image an image captured by an optical chain of saidcamera having a second perspective which is the closest to a firstperspective of an optical chain of said camera that was used to capturethe first reference image.

Returning to 3036, in step 3036 the camera device synthesizes areference image from at least two of said multiple images captured bydifferent optical chain modules during said second time period based onthe detected amount of motion. Operation proceeds from step 3032, viaconnecting node B 3042, to step 3044.

In step 3044 the camera device uses the second reference image and atleast one other image in said first plurality of images to generate acomposite image corresponding to said second time period. In someembodiments, step 3044 includes one or more or all of steps 3046, 3048,3050, 3052, and 3054.

In step 3046, the camera device generates for said at least one otherimage, first image shift information indicating an amount of a firstimage shift between the second reference image and at least one otherimage. Operation proceeds from step 3046 to step 3050. In step 3050 thecamera device performs at least one of a shift, warp or other imagedistortion operation to at least one other image as a function of thefirst image shift amount to generate first modified image data.

In step 3048, the camera device generates for a third image captured byan optical chain of said camera during the second time period, thirdimage shift information indicating an amount of a third image shiftbetween the second reference image and the third image. Operationproceeds from step 3048 to step 3052. In step 3052 the camera deviceperforms at least one of a shift, warp or other image distortionoperation to the third image as a function of the third image shiftamount to generate modified third image data.

Operation proceeds from step 3050 and step 3052 to step 3054, in whichthe camera device combines at least two of said second reference image,said first modified image data, and said modified third image data togenerate said composite image.

Operation proceeds from step 3020 to step 3055, in which the cameradevice stores, displays and/or outputs the generated composite image,e.g., a second composite image corresponding to the second time period.Operation proceeds from step 3055, via connecting node C 3056, to step3058. In step 3058, the camera device updates the current time period byone, e.g., sets T=T+1. Operation proceeds from step 3058 to step 3060.In step 3060 the camera device captures images corresponding to thecurrent image capture time period (T), e.g., the third time period.Operation proceeds from step 3060 to step 3062 in which the cameradevice stores images corresponding to the current image capture timeperiod, e.g., the third time period. Operation proceeds from step 3062to step 3064, in which the camera device generates a composite image forthe current image capture time period, e.g., the time period. Step 3064includes steps similar to those as described previously for step 3020.For example for generating a composite image for the third time period,step 3064 includes: a step including a detection for an amount of motionbetween the third time period and the second time period, a stepincluding producing a third reference image from a plurality of imagescaptured by different optical chains of said camera device during thethird time period, and a step for using the third reference image and atleast one other image in a plurality of image captured during the thirdtime period to generate a composite image corresponding to the thirdtime period.

Operation proceeds from step 3064 to step 3065 in which the cameradevice stores, displays and/or outputs the generated composite image,e.g., the generated third composite image. Operation proceeds from step3065, via connecting node C 3066, to step 3058.

FIG. 18, comprising the combination of FIG. 18A, FIG. 18B, FIG. 18C andFIG. 18D, is an assembly of modules 3100, comprising the combination ofPart A 3199, Part B 3197, Part C 3195 and Part D 3193, in accordancewith an exemplary embodiment.

In one embodiment the assembly of modules 3100 shown in FIG. 18 is partof or used in place of the assembly of modules 2818 of camera device2800 of FIG. 15. The modules in the assembly 3100, when executed by theprocessor 2810 control the camera device 2800 in one embodiment toimplement the method described with regard to FIG. 17. While the modulesof assembly of modules 3100 of FIG. 18 may, and in some embodiments areimplemented using software, in other embodiments they are implemented inhardware, e.g., as circuits, which may and in some embodiments areincluded in the camera device 2800, e.g., as assembly of modules 2880.

In another embodiment, some of the modules of assembly of modules 3100are included as part of assembly of modules 2818 and/or assembly ofmodules 2880 of camera device 2800 and some of the modules of assemblyof modules 3100 are included as part of assembly of modules 1418 ofcomputer system 1400. For example, in one exemplary embodiment, imagedata collection including the capturing of images from multiple opticalchains, and camera motion measurements which are performed by gyroscopesand accelerometers, are be performed by the camera device 2800, andprocessing of the collected captured image data and processing ofcollected camera motion measurements is performed by computer system1400, which generates reference images and composite images. In anotherexemplary embodiment, image data collection is performed by the cameradevice 2800, which captures images from multiple optical chains, andprocessing of the collected captured image data including determinationof motion based on images is performed by computer system 1400. In otherembodiments, some steps of the processing are performed by the cameradevice 2800 and other steps of the processing are performed by thecomputer system 1400.

Assembly of modules 3100 includes a module 3103 configured to monitorfor camera motion, e.g., using gyroscopes and accelerometers and amodule 3105 configured to store motion information corresponding tocaptured images, e.g., information indicating motion between a lastimage capture time period (T−1) and the current image capture timeperiod (T) with images captured in the current image capture timeperiod.

Assembly of modules 3100 further includes a module 3104 configured toinitialize the current image capture time period (T) to 1, a module 3106configured to capture images during a first image capture time period(T=1) using multiple optical chains of the camera device, a module 3108configured to store images captured during the first image capture timeperiod, a module 3109 configured to produce a first reference image,e.g., select the first reference image as the image corresponding to oneof the optical chains, e.g., the center optical chain or the cameradevice. Assembly of modules 3100 further includes a module 3110configured to generate a composite image from images captured during thefirst time period, and a module 3011 configured to store, display,and/or output the generated composite image.

Assembly of modules 3100 further includes a module 3012 configured toupdate the current image capture time period by one, e.g., set T=T+1; amodule 3114 configured to capture image corresponding to the currentimage capture time period (T), e.g., the second image capture timeperiod, and a module 3116 configured to store images corresponding tothe current image capture time period, e.g., the second time period.

Assembly of module 3100 further includes a module 3120 configured togenerate a composite image for the current image capture time period,e.g., the second time period. Module 3120 includes a module 3122configured to detect an amount of motion, a module 3132 configured toproduce a second reference image, from a first plurality of imagescaptured by different optical chains of said camera device during thesecond time period, and a module 3144 configured to use the secondreference image and at least one other image in said first plurality ofimages to generate a composite image corresponding to the second timeperiod. Module 3122 includes a module 3124 configured to detect anamount of motion of a camera device including multiple optical chains,e.g., motion between said first time period and the second time period.In some embodiments, module 3124 uses the stored motion information frommodule 3105, e.g., based on gyro and accelerometer measurements, todetect an amount of motion. Module 3122 includes module 3126 configuredto detect an amount of motion between an image corresponding to a secondtime period and an image corresponding to a first time period. Module3126 includes a module 3128 configured to compare the content of animage corresponding to the first time period to an image correspondingto a second time period. Module 3128 includes a module 3130 configuredto compare a first image captured by a first optical chain module of thecamera during the first time period to a second image captured by saidfirst optical chain module during the second time period.

Module 3132 configured to produce a second reference image includes amodule 3134 configured to select a reference image from images capturedby different optical chains of said cameras during the second timeperiod based on the detected amount of motion and a module 3136configured to synthesize a reference image, the second reference image,from at least two of the multiple images captured by different opticalchain modules during the second time period based on the detected amountof motion. Module 3134 includes a module 3138 configured to select thereference image based on the detected motion of the camera including aplurality of optical chain modules having different perspectives,different ones of said plurality of images being captured by differentones of said plurality of optical chain modules. Module 3138 includes amodule 3139 configured to select as the second reference image an imagecaptured by an optical chain module of said camera having a secondperspective which is the closest to a first perspective of an opticalchain module of said cameras that was used to capture the firstreference image.

Module 3134 includes a module 3140 configured to select the secondreference image based on detected motion in images captured by theoptical chain modules of the camera device. Module 3140 includes amodule 3141 configured to select as the second reference image an imagecaptured by an optical chain of said camera having a second perspectivewhich is the closet to a first perspective of an optical chain of saidcamera that was used to capture the first reference image.

Module 3144 includes a module 3146 configured to generate for at leastone other image, first shift information indicating an amount of a firstimage shift between the second reference image and at least one otherimage, a module 3148 configured to generate for a third image capturedby an optical chain of said camera during said second time period, thirdimage shift information indicating an amount of a third image shiftbetween the second reference image and said third image. Module 3144further includes a module 3150 configured to perform at least one of ashift, warp or other image distortion operation to at least one otherimage as a function of said first image shift amount to generate firstmodified image data, a module 3052 configured to perform at least one ofa shift, warp or other image distortion operation to said third image asa function of said third image shift amount to generate modified thirdimage data. Module 3144 further includes a module 3154 configured tocombine at least two of said second reference image, said first modifiedimage data, and said modified third image data to generate saidcomposite image.

Assembly of modules 3100 further includes a module 3155 configured tostore, display, and/or output the generated composite image, e.g., thegenerated second composite image. Assembly of modules 3100 furtherincludes a module 3158 configured to update the current image capturetime period, e.g., set T=T+1, a module 3160 configured to capture imagescorresponding to the current image capture time period (T), e.g., thethird time period, a module 3162 configured to store imagescorresponding to the current image capture time period, e.g., the thirdtime period, a module 3164 configured to generate a composite image forthe current image capture time period, e.g., the third time period, anda module 3165 configured to store, display, and/or output the generatedcomposite image, e.g., the third composite image.

Numerous additional variations and combinations are possible whileremaining within the scope of the invention. Methods and apparatus whichuse multiple optical chains to capture multiple images of an area at thesame time are described. The multiple captured images may, and in someembodiments are then combined to form a combined image. The combinedimage in various embodiments is normally of higher quality than would beachieved using just a single one of the optical chains. The use ofoptical chains at multiple times can be used to capture sets of imageswhich are then processed with each set being used to generate acomposite image, e.g., frame, corresponding to the image capture timeperiod. Motion between image capture time periods, e.g., image capturestart times, for successive frames can be detected depending on theembodiment using gyroscopes, accelerometers and/or other devices. Outputof such devices and/or information indicating motion of the cameradevice detected since the last frame, e.g., image capture time, can andsometimes is stored with the captured images, e.g., frames. In someembodiments rather than monitor for camera motion using accelerometersor other devices during the time period in which a video or other imagesequence is captured, the images are stored and then imagescorresponding to consecutive time periods are compared to detect motionof the camera device. For example, the image captured by a camera moduleat time T1 may, and in some embodiments is compared to an image capturedby the same camera modules at a second later time T2. Comparison of oneor more pixels, e.g., corresponding to a stationary object in the imagecan and in some embodiments is used to detect camera motion. In someembodiments, the motion of a camera may be predictable and/or intendedas is sometimes the case in a movie sequence where a camera isintentionally moved at a slow and predictable rate along a predeterminedpath. In such embodiments known intentional motion may be subtractedfrom a detected amount of motion before taking image stabilizationaffects or selecting a reference image or generating a synthesized imageto be used as a reference frame. Accordingly, in at least some suchembodiments intended motion may result in changes to the image capturedbut unintended motion that may unintentionally affect the perspective ofone or more camera modules may be, and sometimes is, compensated for bythe selection of a reference image corresponding to a differentperspective than the previously used reference image or by generating asimulated reference image from a perspective which takes intoconsideration the unintended camera motion since the preceding imagecapture time period.

In some embodiments a method is used which involves Capturing image at atime t1 using multiple optical chains of a camera device where thedifferent optical chains each include a lens and sensor, capture imagesat a later time t2 using the optical chain modules. T1 and t2 maybe thetime at which the image capture implemented by the optical chain modulesbegins at each of two sequential image, e.g., frame, capture times.Gyroscopes and/or accelerometers are used to detect forces on the cameraover time in some embodiment and the output is used in some cases todetect camera motion between t1 and t2. The motion information maybe,and in some embodiments is, stored in memory and/or output with theimages captured by the different optical chain modules of the cameradevice at time T2. The particular method may, and sometimes does furtherinvolve detecting an amount of motion, e.g., camera device motion, fromt1 to t2 based on either 1) gyro/accelerometer output or 2) a comparisonof an image corresponding to time period t2 to an image corresponding totime t1. In one such embodiment a reference image, e.g., frame, is thengenerated based on the determined amount of motion. The reference frameis generated either by: i) selecting one of the captured imagescorresponding to t2 based on the detected amount of motion, e.g. whenthe motion indicates the perspective has changed by an amountcorresponding to the difference between the perspective of two opticalchains or ii) synthesizing a reference image from two images capturedduring time period T2, e.g., in the case where the detected amount ofmotion indicates a shift, e.g., an unintended shift, in perspectivewhich is between the difference in the amount of perspective of twooptical chains of the camera device rather than matching a difference inperspective between two optical chains. When a reference frame used inT2 is from a different optical chain which provided the reference framefor T1, there is a switch in the optical chain of the camera devicewhich is being used to provide reference frames. Consider for examplethe case where the center optical chain is used to supply an initialreference frame and then due to motion the center camera module nolonger corresponds to the center of the desired scene being captured dueto unintended motion but another, e.g., second camera modules of thecamera device has a perspective corresponding to the center of the scenearea of interest. In such a case a switch would be made from using theimage provided by the center camera module as the reference frame tousing the image provided by the second camera module as the referenceframe continues to have the desired perspective despite the motion ofthe camera. Once the appropriate reference frame is generated by frameselection or synthesization, the reference frame is then used togenerate a composite image for time period t2. For example, thereference frame may be used to determine image cropping or other imagemodifications to be applied to images captured by other camera chains attime T2 before the images are combined to generate the composite image.The composite image generation process may, and in some embodimentsdoes, involve generated a pixel value for the composite image from pixelvalues of multiple different images captured at time T2.

The process may be repeated with the reference image used at time T2being used as the reference image of the preceding time period when inimage for the next image capture time t3 is to be generated with theprocess repeating in a similar manner for subsequent image capture timeperiods in at least one embodiment.

While the term image is used in many locations the term frame is usedinterchangeably at various locations in the present application. Eachimage or frame is normally represented, i.e., comprises, multiple pixelvalues which can be stored in memory and/or communicated to anotherdevice for processing along with associated movement information inthose embodiments where motion information detected in the camera foruse in image processing.

Image capture is performed by the camera device including multipleoptical chains but composite image generation can be performed by thecamera device or by another device after the image capture process

Various embodiments, provide many of the benefits associated with use ofa large lens and/or large high quality sensor, through the use ofmultiple optical chains which can normally be implemented using smallerand/or lower cost components than commonly used with a high qualitylarge lens single optical chain camera implementation.

In various embodiments an optical chain, e.g., camera module, includes acombination of elements including one or more lenses, a lightredirection device and a sensor. The light redirection device is a lightdiverter and may take various forms, e.g., it may be a mirror or prism.The light redirection device may be hinged to allow the angle and thusdirection in which an optical chain is pointing to be changed by movingthe light redirection device.

In at least some embodiments images captured by different optical chainswith non-round apertures having different orientations are combined. Insome embodiments the images from two, three or more, e.g., six or more,optical chains with different orientations are combined to form a singlecombined image. While images from optical chains with differentorientations are combined in some embodiments, it should be appreciatedthat images captured by more than one optical chain with the sameorientation can be combined with one or more images captured by opticalchains with a different orientation, e.g., relative to the bottom of thecamera, e.g., the horizontal, for purposes of explanation. Thus, bycombining images from different optical chains many advantages can beachieved allowing for multiple small lenses to be used and a relativelythin camera housing as compared to systems using a single large roundlens.

In various embodiments the outer lens of the multiple optical chains arefixed and thus unlike many conventional zoom camera devices in suchembodiments the outer lenses, i.e., the lenses on the face of thecamera, do not move out of the camera body and are fixed with respect tothe face of the camera even during zoom operations. The outermost lensesmay, and in some embodiments do have zero or very little optical powerand serve as a cover to keep dirt out of the optical chains to which theouter lens corresponds. Thus, the entry of an optical chain may becovered by a clear cover as opposed to a lens with an optical power. Theouter lens in such embodiments may be implemented using flat glass orplastic. In some embodiments a slideable cover is slide over the outerlenses when the camera is to be placed in storage and slide back whenthe camera device is to be used. FIG. 14 shows one such embodiment withthe lenses being uncovered and the cover slide to a position in whichthe case which includes the lens cover can be used as a camera grip orhandle.

In some embodiments while a portion of the outermost lens may extendfrom the front of the camera device beyond the surface of the cameradevice, the outermost lenses generally extend, if at all, a small amountwhich is less than the thickness of the camera. Thus even during use thelenses to not extend significantly beyond the face of the camera devicein which the optical chains are mounted and normally less than half thethickness of the camera device at most.

In many if not all cases images representing real world objects and/orscenes which were captured by one or more of the optical chain modulesof the camera device used to take the picture are preserved in digitalform on a computer readable medium, e.g., RAM or other memory deviceand/or stored in the form of a printed image on paper or on anotherprintable medium.

While explained in the context of still image capture, it should beappreciated that the camera device and optical chain modules of thepresent invention can be used to capture video as well. In someembodiments a video sequence is captured and the user can select anobject in the video sequence, e.g., shown in a frame of a sequence, as afocus area, and then the camera device capture one or more images usingthe optical chain modules. The images may, and in some embodiments are,combined to generate one or more images, e.g., frames. A sequence ofcombined images, e.g., frames may and in some embodiments is generated,e.g., with some or all individual frames corresponding to multipleimages captured at the same time but with different frames correspondingto images captured at different times.

Different optical chain modules maybe and sometimes are controlled touse different exposure times in some embodiments to capture differentamounts of light with the captured images being subsequently combined toproduce an image with a greater dynamic range than might be achievedusing a single exposure time, the same or similar effects can and insome embodiments is achieved through the use of different filters ondifferent optical chains which have the same exposure time. For example,by using the same exposure time but different filters, the sensors ofdifferent optical chain modules will sense different amounts of lightdue to the different filters which allowing different amount of light topass. In one such embodiment the exposure time of the optical chains iskept the same by at least some filters corresponding to differentoptical chain modules corresponding to the same color allow differentamounts of light to pass. In non-color embodiments neutral filters ofdifferent darkness levels are used in front of sensors which are notcolor filtered. In some embodiments the switching to a mode in whichfilters of different darkness levels is achieved by a simple rotation ormovement of a filter platter which moves the desired filters into placein one or more optical chain modules.

The camera devices of the present invention supports multiple modes ofoperation and switching between different modes of operation. Differentmodes may use different numbers of multiple lenses per area, and/ordifferent exposure times for different optical chains used to capture ascene area in parallel. Different exposure modes and filter modes mayalso be supported and switched between, e.g., based on user input.

Numerous additional variations and combinations are possible whileremaining within the scope of the invention. Cameras implemented in someembodiments have optical chains which do not extend out beyond the frontof the camera during use and which are implemented as portable handheldcameras or devices including cameras. Such devices may and in someembodiments do have a relatively flat front with the outermost lens orclear, e.g., (flat glass or plastic) optical chain covering used tocover the aperture at the front of an optical chain being fixed.However, in other embodiments lenses and/or other elements of an opticalchain may, and sometimes do, extend beyond the face of the cameradevice.

In various embodiments the camera devices are implemented as digitalcameras, video cameras, notebook computers, personal data assistants(PDAs), or other portable devices including receiver/transmittercircuits and logic and/or routines, for implementing the methods of thepresent invention and/or for transiting captured images or generatedcomposite images to other devices for storage or display.

The techniques of the present invention may be implemented usingsoftware, hardware and/or a combination of software and hardware. Thepresent invention is directed to apparatus, e.g., dedicated cameradevices, cell phones, and/or other devices which include one or morecameras or camera modules. It is also directed to methods, e.g., methodof controlling and/or operating cameras, devices including a camera,camera modules, etc. in accordance with the present invention. Thepresent invention is also directed to machine readable medium, e.g.,ROM, RAM, CDs, hard discs, etc., which include machine readableinstructions for controlling a machine to implement one or more steps inaccordance with the present invention.

In various embodiments devices described herein are implemented usingone or more modules to perform the steps corresponding to one or moremethods of the present invention, for example, control of image captureand/or combining of images. Thus, in some embodiments various featuresof the present invention are implemented using modules. Such modules maybe implemented using software, hardware or a combination of software andhardware. In the case of hardware implementations embodimentsimplemented in hardware may use circuits as part of or all of a module.Alternatively, modules may be implemented in hardware as a combinationof one or more circuits and optical elements such as lenses and/or otherhardware elements. Thus in at least some embodiments one or moremodules, and sometimes all modules, are implemented completely inhardware. Many of the above described methods or method steps can beimplemented using machine executable instructions, such as software,included in a machine readable medium such as a memory device, e.g.,RAM, floppy disk, etc. to control a machine, e.g., a camera device orgeneral purpose computer with or without additional hardware, toimplement all or portions of the above described methods, e.g., in oneor more nodes. Accordingly, among other things, the present invention isdirected to a machine-readable medium including machine executableinstructions for causing or controlling a machine, e.g., processor andassociated hardware, to perform e.g., one or more, or all of the stepsof the above-described method(s).

While described in the context of an cameras, at least some of themethods and apparatus of the present invention, are applicable to a widerange of image captures systems including tablet and cell phone deviceswhich support or provide image capture functionality.

Images captured by the camera devices described herein may be real worldimages useful for documenting conditions on a construction site, at anaccident and/or for preserving personal information whether beinformation about the condition of a house or vehicle.

Captured images and/or composite images maybe and sometimes aredisplayed on the camera device or sent to a printer for printing as aphoto or permanent document which can be maintained in a file as part ofa personal or business record.

Numerous additional variations on the methods and apparatus of thepresent invention described above will be apparent to those skilled inthe art in view of the above description of the invention. Suchvariations are to be considered within the scope of the invention. Invarious embodiments the camera devices are implemented as digitalcameras, video cameras, notebook computers, personal data assistants(PDAs), or other portable devices including receiver/transmittercircuits and logic and/or routines, for implementing the methods of thepresent invention and/or for transiting captured images or generatedcomposite images to other devices for storage or display.

The techniques of the present invention may be implemented usingsoftware, hardware and/or a combination of software and hardware. Thepresent invention is directed to apparatus, e.g., dedicated cameradevices, cell phones, and/or other devices which include one or morecameras or camera modules. It is also directed to methods, e.g., methodof controlling and/or operating cameras, devices including a camera,camera modules, etc. in accordance with the present invention. Thepresent invention is also directed to machine readable medium, e.g.,ROM, RAM, CDs, hard discs, etc., which include machine readableinstructions for controlling a machine to implement one or more steps inaccordance with the present invention.

Numerous additional embodiments are possible while staying within thescope of the above discussed features.

What is claimed:
 1. A method of generating images, the methodcomprising: detecting an amount of motion, said detected amount ofmotion being a detected amount of motion of a camera device includingmultiple optical chains or a detected amount of motion between an imagecorresponding to a second time period and an image corresponding to afirst time period; producing a second reference image, from a firstplurality of images captured by different optical chains of said cameradevice during the second time period, producing a second reference imageincluding at least one of: i) selecting a reference image from imagescaptured by different optical chain modules during said second timeperiod based on the detected amount of motion or ii) synthesizing areference image from at least two of said multiple images captured bydifferent optical chain modules during said second time period based onthe detected amount of motion; and using the second reference image andat least one other image in said first plurality of images to generate acomposite image corresponding to said second time period.
 2. The methodof claim 1, wherein said first composite image is an image having thesame perspective as said first reference image.
 3. The method of claim1, wherein selecting a second reference image is based on detectedmotion of a camera including a plurality of optical chain modules havingdifferent perspectives, different ones of said first plurality of imagesbeing captured by different ones of said plurality of optical chainmodules.
 4. The method of claim 3, wherein said selecting includesselecting as said second reference image an image captured by theoptical chain of said camera having a second perspective which is theclosest to a first perspective of an optical chain of said camera thatwas used to capture the first reference image.
 5. The method of claim 1,wherein selecting a second reference image is based on detected motionin images captured by optical chain modules of said camera, said motionbeing detected by comparing the content of an image corresponding tosaid first time period to an image corresponding to said second timeperiod.
 6. The method of claim 5, wherein comparing the content of animage corresponding to said first time period to an image correspondingto said second time period includes comparing a first image captured bya first optical chain module of said camera during a first time periodto a second image captured by said first optical chain module duringsaid second time period.
 7. The method of claim 6, wherein saidselecting includes selecting as said second reference image an imagecaptured by the optical chain of said camera having a second perspectivewhich is the closest to a first perspective of an optical chain of saidcamera that was used to capture the first reference image.
 8. The methodof claim 1, wherein said composite image corresponding to said secondtime period is a frame of a video sequence including composite imagescorresponding to multiple sequential frame times; wherein the methodfurther comprises operating a handheld camera device including saidmultiple optical chains to capture said first plurality of images; andwherein said detecting, producing and suing steps are performed by i) aprocessor included in said handheld camera device or a processor whichreceives said first plurality of images from said handheld cameradevice.
 9. The method of claim 1, wherein using said second referenceimage and at least one other image in said first plurality of images togenerate a composite image corresponding to said second time periodincludes: generating for said at least one other image, first imageshift information indicating an amount of a first image shift betweenthe second reference image and said at least one other image.
 10. Themethod of claim 9, wherein using said second reference image and atleast one other image in said first plurality of images to generate acomposite image corresponding to said second time period furtherincludes: generating for a third image captured by an optical chainmodule of said camera during said second time period, third image shiftinformation indicating an amount of a third image shift between thesecond reference image and said third image.
 11. The method of claim 10,wherein using said second reference image and at least one other imagein said first plurality of images to generate a composite imagecorresponding to said second time period further includes: performing atleast one of a shift, a warp or another image distortion operation tosaid at least one other image as a function of said first image shiftamount to generate first modified image data.
 12. The method of claim11, wherein using said second reference image and at least one otherimage in said first plurality of images to generate a composite imagecorresponding to said second time period further includes: performing atleast one of a shift, a warp or another image distortion operation tosaid third image as a function of said third image shift amount togenerate modified third image data; and combining image datacorresponding to at least two of said second reference image, said firstmodified image data and said modified third image data to generate saidcomposite image.
 13. A camera device comprising: a plurality of opticalchain modules; a module configured to detect an amount of motion; amodule configured to produce a second reference image, from a firstplurality of images captured by different optical chains of said cameradevice during the second time period, producing a second reference imageincluding at least one of: i) selecting a reference image from imagescaptured by different optical chain modules during said second timeperiod based on the detected amount of motion or ii) synthesizing areference image from at least two of said multiple images captured bydifferent optical chain modules during said second time period based onthe detected amount of motion; and a module configured to use the secondreference image and at least one other image in said first plurality ofimages to generate a composite image corresponding to said second timeperiod.
 14. The camera device of claim 13, wherein said moduleconfigured to detect an amount of motion includes a module configured todetect an amount of motion of the camera device including multipleoptical chains, e.g., based on gyroscope and/or accelerometermeasurements.
 15. The camera device of claim 13, wherein said moduleconfigured to detect an amount of motion includes a module configured todetect an amount of motion between an image corresponding to a secondtime period and an image corresponding to a first time period.
 16. Thecamera device of claim 13, wherein said first composite image is animage having the same perspective as said first reference image.
 17. Thecamera device of claim 13, wherein said module configured to select asecond reference image includes a module configured to select the secondreference image based on detected motion of a camera including aplurality of optical chain modules having different perspectives,different ones of said first plurality of images being captured bydifferent ones of said plurality of optical chain modules.
 18. Thecamera device of claim 17, wherein said module configured to select asecond reference image includes a module configured to select as saidsecond reference image an image captured by the optical chain of saidcamera having a second perspective which is the closest to a firstperspective of an optical chain of said camera that was used to capturethe first reference image.
 19. The camera device of claim 1, furthercomprising: a module configured to compare the content of an imagecorresponding to a first time period to an image corresponding to asecond time period; and wherein said module configured to select asecond reference image includes a module configured to select the secondreference image based on detected motion in images captured by opticalchain modules of said camera, said motion being detected by comparingthe content of an image corresponding to said first time period to animage corresponding to said second time period.
 20. A non-transitorycomputer readable media comprising stored instructions which whenexecuted by a processor of an image processing system control the imageprocessing system to: detect an amount of motion, said detected amountof motion being a detected amount of motion of a camera device includingmultiple optical chains or a detected amount of motion between an imagecorresponding to a second time period and an image corresponding to afirst time period; produce a second reference image, from a firstplurality of images captured by different optical chains of said cameradevice during the second time period, producing a second reference imageincluding at least one of: i) selecting a reference image from imagescaptured by different optical chain modules during said second timeperiod based on the detected amount of motion or ii) synthesizing areference image from at least two of said multiple images captured bydifferent optical chain modules during said second time period based onthe detected amount of motion; and generate a composite imagecorresponding to said second time period from the second reference imageand at least one other image in said first plurality of images.